Data Preparation using R

Data Analytics - Data Preparation using R

Data preparation is an important step in data science. Before you can start analysis, we need to ingest the required data and then clean and transform it so that it becomes more easy to perform analytics.

R is an open source and powerful software having reliable packages and libraries to perform data manipulation tasks.

I have downloaded the clinical trials data from ACCT website and used that to show data preparation steps. You can reference the complete code published here in a R Markdown document:

http://rpubs.com/kalehdoo/Data_Preparation

Let's walk through the code and understand how to do it. You may want to open the link mentioned above in a separate tab since I will not mention the code here again and will simply refer to the document. 

You would need to first download the data files from ACCT website download tab and unzip them into a folder in your local directory.

Installing the required packages is the first step. 'dplyr' is an important library package to perform data manipulation and 'lubridate' provides important functions to perform operations with date columns.

Save the directory paths into variables. Notice that the '\' in the path are replaced with '/'. 

read.csv function is used to read the pipe delimited data files into dataframes. Another option is to use read.table function can also be used but I found read.csv works better for me. I encountered errors reading records but read.csv worked without any issues for the same data set.

studies table has lots of columns that I did not require for the analysis, so I created a subset with the required columns only. 

subset.data.frame function allows to select the required columns using 'select' parameter. You can also provide any data filter using 'subset' parameter specifying the filter condition if you want a subset of rows.

I need to create new formula columns into the subset of studies created. mutate function helps to do that.

Another function which I found helpful in doing a wildcard search is 'grepl' function.

summarise function is helpful in creating aggregations.

Hope you will find this useful.

Sponsor Key Performance Indicators - Part 1

Sponsor Performance KPI - Part 1

Sponsors are the key stakeholders in clinical trials. It is important to measure the performance of sponsors to understand the trends in the industry and market. Metrics such as studies registered, studies completed, study results posted, collaboration and study completion duration can be key performance indicators from study conduct perspective. We can gain competitive intelligence about competitor sponsors or find opportunities to collaborate with potential partners. The possibilities are endless. 
Let's see who were top Industry sponsors in terms of studies completed in 2018 ranked by number of completed studies. 

There are 4,017 studies completed by industry sponsors in 2018. Novartis is leading the board with 114 studies completed in 2018, followed by GSK and Pfizer with 96 and 92 studies respectively. The chart also shows the number of results posted by those sponsors in 2018. There may be studies completed in 2018 for which the results are not posted yet, but I am not aware of any requirements for posting results. Another metric is the ration of completed studies to the studies with posted results. There should be a linear relationship since more completed studies would mean more posted results. To just verify that, let's create a scatter plot. There is a strong linear relationship. The R-square value is pretty high. See the plot below with a fitted line. It may be interesting to look at sponsors with very low ratio.
  

Now, let's take a look at what's going on with non-industry sponsors in the same year 2018.

NIH, Mayo clinic and Duke University are the top 3 performers. 

The studies completed numbers are comparable  but the posted results and the ratio is very low when compared to sponsors from industry segment. The slope is 0.13 as compared to 0.3 for industry sponsors. So, we see that industry sponsors are posting more results. 

   
We can see that the industry sponsors(blue) have higher posted results for same levels of completed studies as compared to the non-industry sponsors(red).
I am really trying to think why non-industry sponsors have low results postings than industry sponsors. 
Every study is required to submit the results, generally no later than 12 months of completion.
The clinicaltrials.gov explains what all is included in the results and also mentions few valid reasons when the results are not submitted:
https://clinicaltrials.gov/ct2/about-site/results#DisplayOfResults 

Good news!!! I have recreated sponsor analytics using flexdashboards and plotly R so that you all can interact instead of viewing static jpeg images. The charts are very interactive and you can view individual data points. However, in the analysis, I have only included sponsors that have completed at least 10 studies, to reduce the number of data points as well as the skewness caused by them since there were a large numbers of sponsors within that range. I hope you would enjoy that.
Link to Dashboard:
http://rpubs.com/kalehdoo/SponsorDashboard

Summary -There are a total of 2234 sponsors who have completed at least 10 studies in the past out of which there are 647 (29%) sponsors from Industry and the remaining 1587 (71%) from non-industry.
Non-industry sponsors are further classified into Academic and Hospital based on their names (this may not be 100% correct and you may notice some sponsors classified incorrectly).
Keep reading!!!


    

Linear Regression using R and Python

R and Python

There are times when we not only look at the descriptive analysis but also want to make future predictions based on the past trends. We will look at techniques that we can use to predict the number of studies submitted or registered in future years.

We will see how we can use some of the libraries like pandas, statsmodels and  matplotlib in python.
the python code is available on my github repository here: 
https://github.com/kalehdoo/clintrials/blob/master/ctrials_1.py
I will also try to explain the steps and procedures to perform the analysis later.
Here is the final outcome:
Model Summary:

Studies submitted predicted:
For 2019 : 31,022 
For 2020 : 32,479

The data used for the regression is from 2005-2018.
There is still some time left for 2019 to complete so I will come back next year to compare the 2019 actuals with the predicted numbers here. The actuals for 2019 is 19,990 (data until Aug 2019) which was posted in one of the previous posts here.

I have also shown how to do regression using R programming, and also how to interpret the results. The link below has complete code and the analytics:
http://rpubs.com/kalehdoo/sponsor_analytics


Source data extracted from: https://aact.ctti-clinicaltrials.org