{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Formation South Green 2022 - Structural Variants Detection by using short and long reads \n",
"\n",
"# __DAY 1 : How to map reads against a reference genome ?__ \n",
"\n",
"Created by C. Tranchant (DIADE-IRD), J. Orjuela (DIADE-IRD), F. Sabot (DIADE-IRD) and A. Dereeper (PHIM-IRD)\n",
"\n",
"\n",
"***\n",
"\n",
"# Table of contents\n",
"\n",
"\n",
"[I - Preparing data](#data)\n",
"\n",
"* [Download sequencing data (SR & LR) for Simulated clones](#download)\n",
"\n",
"[II - Mapping Practice](#mapping) \n",
" \n",
"[2.1. Mapping short reads vs a reference with `bwa mem`](#bwamem)\n",
"\n",
" * [ Reference indexation](#refindex)\n",
" * [Run the mapping with `bwa mem`](#bwamem2-cmd)\n",
" * [Convert sam into bam `samtools view`](#samtoolsview)\n",
" * [Sort raw bam file `samtools sort`](#sortbam)\n",
" * [Calculate stats from mapping `samtools flagstat`](#flagstats)\n",
" * [Generate a bam file that contains only the reads correctly paired mapped `samtools view`](#corrmap) \n",
" * [Indexing bam files](#indexbam) \n",
" * [EXERCICE : MAP ALL SR WITH BWAMEM2](#mapallmem)\n",
"\n",
"[2.2. Mapping long reads vs a reference with `minimap2`](#minimapé)\n",
" * [EXERCICE : MAP ALL LR WITH MINIMAP2](#mapallminimap)\n",
"\n",
"[III - Centralize final mapping data into a single bam directory](#reorder)\n",
"\n",
"\n",
"\n",
"***\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# __I - Preparing data__ \n",
"\n",
"### Download sequencing data we will use \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# go to work directory and download data\n",
"cd /home/jovyan/work/\n",
"ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Download sequencing data (SR & LR) for Simulated clones \n",
"\n",
"Before starting, please download special data created for this practical training. Data are available on the from I-Trop server.\n",
"\n",
"Each participant will analyse a Clone, results will be complete in this shared file\n",
"\n",
"To generate Clone data, a 1Mb contig was extracted from chromosome 1 of rice.\n",
"\n",
"20 levels of variation were generated and long reads were simulated for each.\n",
"\n",
"We have introduced different variations (SNP, indel, indel+translocations) and also some contaminations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# download available compressed DATA \n",
"wget --no-check-certificat -rm -nH --cut-dirs=1 --reject=\"index.html*\" https://itrop.ird.fr/sv-training/SV_DATA.tar.gz\n",
"# decompress data\n",
"tar zxvf SV_DATA.tar.gz\n",
"rm SV_DATA.tar.gz"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### List the content of the directory work and check that the directory SV_DATA have been created \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# check data \n",
"ls -l"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### List the content of the directory SV_DATA "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"ls -lRt SV_DATA"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# __II - MAPPING PRACTICE__ "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Read congruency is an important measure in determining assembly accuracy.\n",
"\n",
"Clusters of read pairs or single long reads that align incorrectly are strong indicators of mis-assembly.\n",
"\n",
"Reads mapping is usually the firt step before SNP or Variant calling."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Make a folder for your results "
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [],
"source": [
"mkdir -p ~/work/MAPPING-ILL\n",
"cd ~/work/MAPPING-ILL"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Declare important variables \n",
"\n",
"We are going to set up bash variables with the path to our data.We set a bash variable like this : `var=\"value\"`\n",
"and call it as: `echo $var`\n"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"# REFERENCE \n",
"REF_DIR=\"/home/jovyan/work/SV_DATA/REF/\"\n",
"REF=\"/home/jovyan/work/SV_DATA/REF/reference.fasta\"\n",
"\n",
"# ONT DATA\n",
"ONT=\"/home/jovyan/work/SV_DATA/LONG_READS/Clone${i}.fastq.gz\"\n",
"\n",
"# ILLUMINA DATA\n",
"ILL_R1=\"/home/jovyan/work/SV_DATA/SHORT_READS/Clone${i}_R1.fastq.gz\"\n",
"ILL_R2=\"/home/jovyan/work/SV_DATA/SHORT_READS/Clone${i}_R2.fastq.gz\"\n",
"\n",
"#CLONE NUMBER THAT YOU ARE GOING TO ANALYZE \n",
"i=10 "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Print the variable i, REF, ILL_R1 & ILL_R2"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Clone10 /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"/home/jovyan/work/SV_DATA/SHORT_READS/Clone20_R1.fastq.gz /home/jovyan/work/SV_DATA/SHORT_READS/Clone20_R2.fastq.gz\n"
]
}
],
"source": [
"echo \"Clone${i} $REF\" \n",
"echo $ILL_R1 $ILL_R2"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"-------------\n",
"# 2.1. Mapping short reads vs a reference with `bwa mem` "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this practice, we are going to map short reads against a reference. To know, how well do the reads align back to the reference, we use bwa-mem2 and samtools to assess the basic alignment statistics.\n",
"\n",
"In this exercise, we will use reference.fasta assembly as well ILLUMINA READS from your favorite CLONE.\n",
"\n",
"The tool bwa needs 2 steps: \n",
"- **Reference indexing**: `bwa index reference`\n",
"- **Mapping in itself**: `bwa mem -R READGROUP [options] reference fastq1 fastq2 > out.sam`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reference indexation "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before mapping we need index reference file! Check bwa-mem2 index command line."
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {},
"outputs": [],
"source": [
"cd $REF_DIR"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Indexing reference /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"\n",
"[bwa_index] Pack FASTA... 0.01 sec\n",
"init ticks = 153121145\n",
"ref seq len = 2040002\n",
"binary seq ticks = 57691336\n",
"build index ticks = 565957521\n",
"ref_seq_len = 2040002\n",
"count = 0, 576483, 1020001, 1463519, 2040002\n",
"BWT[1932441] = 4\n",
"CP_SHIFT = 5, CP_MASK = 31\n",
"sizeof CP_OCC = 64\n",
"max_occ_ind = 63750\n",
"ref_seq_len = 2040002\n",
"count = 0, 576483, 1020001, 1463519, 2040002\n",
"BWT[1932441] = 4\n",
"CP_SHIFT = 6, CP_MASK = 63\n",
"sizeof CP_OCC = 64\n",
"max_occ_ind = 31875\n"
]
}
],
"source": [
"echo -e \"\\nIndexing reference $REF\\n\"\n",
"bwa-mem2 index $REF"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check that the indexes have been created "
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"reference.fasta reference.fasta.ann reference.fasta.pac\n",
"reference.fasta.0123 reference.fasta.bwt.2bit.64\n",
"reference.fasta.amb reference.fasta.bwt.8bit.32\n"
]
}
],
"source": [
"ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## => Let's map now but only WITH READS FROM ONLY ONE CLONE \n",
"\n",
"* Go into the directory MAPPING-ILL\n",
"* Create a subdirectory to save the files generated by the mapping step. \n",
"Eg: If you are going to analyze the `clone1`, create the subdirectory `dirClone1`. "
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
">>>>>>>>>> Creation directory for Clone10\n",
"\n"
]
}
],
"source": [
"cd ~/work/MAPPING-ILL\n",
"echo -e \"\\n>>>>>>>>>> Creation directory for Clone$i\\n\"\n",
"mkdir -p dirClone$i\n",
"cd dirClone$i"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run the mapping with `bwa mem` "
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
">>>>>>>>>> Mapping Clone10\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 43560000, nseq: 290400\n",
"\t[0000][ M::kt_pipeline] read 290400 sequences (43560000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 290400\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 87298, 134, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (379, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (309, 484)\n",
"[M::mem_pestat] mean and std.dev: (396.64, 25.85)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (274, 519)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (134, 190, 252)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 488)\n",
"[M::mem_pestat] mean and std.dev: (211.16, 91.90)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 606)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 290400 reads in 40.848 CPU sec, 5.331 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.209537 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.55 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.90, (0.90, 0.90)\n",
"\tWriting IO time (SAM) avg: 0.29, (0.29, 0.29)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.53\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.33, (5.33, 5.33)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.72, (1.72, 1.72)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.58, (3.58, 3.58)\n",
"\t\tSMEM compute avg: 1.43, (1.48, 1.38)\n",
"\t\tSAL compute avg: 0.17, (0.23, 0.13)\n",
"\t\tBSW time, avg: 1.91, (1.93, 1.90)\n",
"\n",
"\tTotal allocs: 299305 = 299305 out total requests: 351940, Rate: 0.85\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n"
]
}
],
"source": [
"echo -e \"\\n>>>>>>>>>> Mapping Clone$i\\n\"\n",
"bwa-mem2 mem -M -t 8 $REF $ILL_R1 $ILL_R2 > Clone$i.sam"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"### Check that the file `.sam` have been created by `bwa mem` \n"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 161216\n",
"-rw-r--r-- 1 jovyan users 18213243 Jun 21 21:03 Clone10.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 120830158 Jun 21 21:13 Clone10.sam\n",
"-rw-r--r-- 1 jovyan users 26030273 Jun 21 21:03 Clone10.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 381 Jun 21 21:03 Clone10.sorted.flagstat\n"
]
}
],
"source": [
"ls -l"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Display the first and the end of the sam file just created "
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"@SQ\tSN:Reference\tLN:1020001\n",
"@PG\tID:bwa\tPN:bwa\tVN:2.0pre2\tCL:bwa-mem2 mem -M -t 8 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/SHORT_READS/Clone20_R1.fastq.gz /home/jovyan/work/SV_DATA/SHORT_READS/Clone20_R2.fastq.gz\n",
"Reference-Clone20290400\t77\t*\t0\t0\t*\t*\t0\t0\tATCACATTGTATACGTAGCTGTATTCGCCCCCATTACGTTTACCATCACTTCCACTGCACGATAACTGGGCATTCGCACCTGTTACGGAGTCGAGGCTACTTGAGTTACTTACATGGACTGCGGCATACGAAGCACTTAACTCTAGGGTG\t=CCGGGGGGGGGGIIIGIIIIIII5IICGIIGIIGIIIGICGCIGICGIGICGGIGGGGG88IG8CCIIGGIGGG5CGGGGC5GGGG=8CG=CCGCGGCG=GCGG5GGGCGCGGGGGGGG8GGGCCGGGGC5GGGGGGCCGCGG5GGGCC\tAS:i:0\tXS:i:0\n",
"Reference-Clone20290400\t141\t*\t0\t0\t*\t*\t0\t0\tATGGCCACCCTGCTACTGGCAATGATAGTACGGCTGCTCAGAGTCTTAGACCACGTCTTGGAGAGACAGACCGACACTCCGATAATGCTAACGTTTCGAGCGCTCAGTTGGAAGTCACTAGCATTAAAGAATGGATAACCACCACCAAAG\tCCC=GGGGGGGGG5=GIIIIIIIIGIIGIIIICIIIIGI=GCIICIIGCIG=GCGI5GCGCIGGIC8GGIGGC=GCGCGI==8GCGGCGGG8G=GGCG=GGCCIIIIGCGGGGGCG=GCGGGGG5CCCGCGCGGGGGG=GCGC==C=CCC\tAS:i:0\tXS:i:0\n",
"Reference-Clone20290398\t99\tReference\t934278\t60\t150M\t=\t934525\t397\tAGAGAAGTGGATATTCGAGAGTGAGGTTGGGGCACTAAGGCAGCGTATTCCCGAAGGTCCTCGTAAATGGCTTGAAGACGAGTTGTTGGATAACGATAAGTGGTCTCGTGCATACGATAGAAATGGCCGCCGGTGGGGTTACATCACAAC\tCCCGGG=G8GGGGIIIGIIIGGIIGIGGIIIICIG8IGIIGGCII8I8I=GIG5IGGIIGIGIIIGIGGGII8IGG5CGGCCGGCCCGCGGCGG=GC=GGIGGGGGGGGGCGGGGCGGGCGGGGGGCGG=GGGGCGCG=GGCGGGCGGGC\tNM:i:8\tMD:Z:0C14T5G2T4A48T14A50G5\tAS:i:114\tXS:i:0\n",
"Reference-Clone20290398\t147\tReference\t934525\t60\t150M\t=\t934278\t-397\tTTTCTGAACACACAGGAATCAGGGAAGCCAGTACAGTTAGGGGAGCGTTGGTCCACCAAGGTTGCTAGTAAGATGGTGGTGCAAAAAAGTAAGGCGAACATACACACTTCACGGTATTTCGATAAGCACAAAAAGACATATGAGGTCACT\tGGGGGCCG==G8CGGGG8CCGCG5G5CGGG=CGGCG=CC8GCCIIICIC8GGCGGGCGCGGGGCGCG8GGG5GGCGGGIGGIIGGCCICIGGIGIGGIGG8IIIGII5I8G8IIII5GCI=IIGC8GCIIICIICGGGGGGGGGGGG=CC\tNM:i:14\tMD:Z:3G6G3T3G3T5G46A0A1A21A10A3G12G20A0\tAS:i:86\tXS:i:0\n",
"Reference-Clone20290396\t81\tReference\t324007\t0\t150M\t=\t357781\t33626\tCTCGAAAGAGCCGTATGTGAAAATTAACTTGTACGCGGGGTCATGAAATTCGTACCCCATGCAAATCCTTAATTTTTATAGGCCCTTGGTAGTACAGTTGATCGATTCAAATAAATGCTGCCGAAAATAGGTTACGTTTTCTTTACCAGA\t=CGGGCCCGCGGGGGCGGCG8GGC5GGGGCGGCGGGGGGGG=GG=GGGGIGGGCGCCGCGCG5G5GGGCGIG5CGCICGIGCCG=C5GCCIGIGGGGGIIIIIIGGIIIGICCCIIIIGIGICIGGIIIC=IGIGIIGGGGGGGCGGCCC\tNM:i:0\tMD:Z:150\tAS:i:150\tXS:i:150\tXA:Z:Reference,-390313,150M,0;Reference,-357160,150M,0;\n",
"Reference-Clone20290396\t161\tReference\t357781\t0\t150M\t=\t324007\t-33626\tTTGGGTTTTGTTTAGTTGAATTTTGCTAAGTGCCATTCACCTTGCCTCTAGTCCTCGCTCGATTAGTCGGAGAATATAGATTCAGAGCCCCCAATAGTTGGTGTGTTGGTTTGCCGGGTCGATTAGATCTACCACTTCAATCGCAACTAT\tCCCGGGGGGGGGGGCIIIGGCIIGIIGIIII5IGGCIIIII=GGGGG8IGC8CIG=GGGICGI=GCCIIGGGGICCGGCC=GCCCGCGGGGGCCGGGGGGGCCIIIIGC8CGGGGGC8CG8GCCGGCGG=CGGGC=C8GG=CCGGGC8GC\tNM:i:5\tMD:Z:26C17T25C2C34A41\tAS:i:125\tXS:i:125\tXA:Z:Reference,+324628,150M,5;Reference,+390934,150M,5;Reference,+360903,150M,7;Reference,+327750,150M,7;Reference,+394057,150M,7;\n",
"Reference-Clone20290394\t99\tReference\t87570\t60\t150M\t=\t87838\t418\tAAAAGCTAATAAGATGACTCGCTCTGTAAGATAGCAGCAGCTTAATGGATGGGTCACTGCACGTCAGTCTTCAGCTGGTTCCTCCGGCCTCCAGATAGCAATATCCGAAAATACACAATTATGAAACTTATAACCATCAAATAGCTCAAG\tCCC=GGGGGCGGGIGIIGGIGGGGGIGIGGIGGIIIIGICI5GCIIICICIGGIGGIGIIIGGICGGICC=GCGGGCG5=GGC8GGGGGGCC=GCCGGCGICCCGG=GG=GCGCCC=GGGGGGGGGGGGGCGCGGGCGGGGGGGGCG55C\tNM:i:11\tMD:Z:5A8G4A13C3A24T30T11T16A14G11C0\tAS:i:99\tXS:i:0\n",
"Reference-Clone20290394\t147\tReference\t87838\t60\t150M\t=\t87570\t-418\tTAATTTGAGGGAAATTTCGTAACATATAGAACCAAACAAAAAACAAGGTTAGGGTATCAGAATTTATTACCTTATGTCAGATTGAATTTTTATCGAGTCAATGTTGTTGAGCACAGAGCACATATATGGCAACTCATACTTTTTAAGACG\tCGC8CGGGGCGCGGGGGCC=GGGCGGGCGGCGCGGGGGGGCGGI8IICCGCC8=GGC55CCGGCGGG==8CGC=CGG5CCG=GGIGGIGCC5G8IGIGIIICCIGGIII=IIGIGI=GIIIGGIGGIGIIIIIIGIIGCGGGGGGGGCCC\tNM:i:17\tMD:Z:6C3A1C12G1C21A2A1A1A9A3T6G1T13A0A1C50T2\tAS:i:69\tXS:i:0\n"
]
}
],
"source": [
"head Clone10.sam"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reference-Clone2010\t99\tReference\t480500\t60\t150M\t=\t480734\t384\tCCGGGCCGAGCCGCCCCAGCCACCGCCACCCGAAGACAAGTGGAGTGCAAAGAAAGTGAGTTGGAAAGAAAGAGAGTGTGAGGGGAGCGGATAGAAAGAGAGCGAGAATGTGTAGGTGAGTGGAGGTAAAATGTCTGGATAGGGGAGTAA\tCCCGGGGGGGCGGIGGIIGGGIIIIGIIGIIGIGIGIICGGCCIIGIGICIIIGGGIG8=IGGGICGGGICC5CGGGGCGGGGGGGGCCGCC8GGGGGGGIGGG=GGGCGCCGGGGGGC5GCCGGCGGGGCGGGGGCCCG5GGGGGGCCG\tNM:i:13\tMD:Z:27G8G0G9G1G6A19A10A14A4G5G0T17A17\tAS:i:85\tXS:i:0\n",
"Reference-Clone2010\t147\tReference\t480734\t60\t150M\t=\t480500\t-384\tTTAACAGATAAGAAATTCAAATACTATTATGAAGCAATAAACGATTTCAAATGGAAAAGTTGTGAACTACAAAGTTGTATAACTTAGCAAGTTCTACAACTTTTGTTTTGATCATTTTCTATAGGAATTTGTTTGAACTGTTTGAATTTG\tG=GC=CCGGGCGGCCG===CC8GCGGGGCGGC5G5CGCGGCGGICCIIG5CCCCGCCGGGG8CGGGG8GGCGGGIGGGGGGIG=IIGGIIG=CGIIICIG=III=CGICGCI5IIIII8IIIGIIIIICGICIIIGIGGGGGGGG=GC=C\tNM:i:8\tMD:Z:4A1T16A4T12T44T36T2C23\tAS:i:113\tXS:i:0\n",
"Reference-Clone208\t83\tReference\t857144\t60\t150M\t=\t856924\t-370\tCGAGGCCGCCGCCATGCGGCGCGATCGTCACGGGCTCGCGCTCCAGTCGTTCAACGCGCGGGCTCGCGTGTACATCGAGGAGTCCGGCCTCGTCGCCCTAATCAAGTGAGGATGCAGCATTGAAGTTTTTCATTTATCTACTGTGTAATA\tGC8CGG=GGGGGCGGGG=GGCGGCCGGGCCGGCCCCGGGG8GGGCGC8GIG=G=GGGCGGCCC=CGCGGGGGCGCCGGGGGGG=GGIIIGG=IGIIIII=GIGCIIIC=GGGIIGIIIGIIIG=IGGIGIIGCIIIIGGGGGGGGGCCCC\tNM:i:12\tMD:Z:24C2C1C2A8A21G36G17T4T7G5A4T7\tAS:i:90\tXS:i:0\n",
"Reference-Clone208\t163\tReference\t856924\t60\t150M\t=\t857144\t370\tCAGTCCCCTCCTCCCCCACGCTCTCGGGCACTACGGACGGCGGTCTCGCCGCGGCGGCTGCGGCGGCTGACGACGCCGCCCTCCCGCGCGCGCTCTAGGTGCGGCGCGCGTTGGCCGGGGAGGGGCTTGTGGCGGCGCTCCGAGAGGGGA\tCC=G5GGGGGGG5IIIGGGIGIICIGC5GGIIIIGIII5IGGCGIIGG=G8GCGGIIGIGIGCGGGGGGGCGGGCGCG=CG=G=GGGGGG=CGCG5GCGCGGICC5CGGCGCCCG=CCGG8GGGCGGGGG=GGGG8G=GCCCGGGCGGGG\tNM:i:15\tMD:Z:1C21C3A4G25C4T3G10T3G1G10G14G12T20G0C4\tAS:i:82\tXS:i:0\n",
"Reference-Clone206\t77\t*\t0\t0\t*\t*\t0\t0\tATTGCCTAAGTCGCAGTCTCGTTCACACGACCCCTAACCGGGCCGGTAGAACCCTTTATTAATTGAGTGAGTTGTATAACTTCCAACTTGCGCAACCCGCTTGTGAGGATGGGCCCCATAACGTATACTTGGGTTCTTACCCACTATCAC\tCCCCGGGG5GGGGICIGICII5GII5GGIGGIGII=IIIIIGIIIICIGGGGCGIIGGIGG=IGGGGGGGGGGCCCGC8CGCGGGGGCGCGCCG85CGGGCGCGCGG8G8GCGCGCGCCGGG=GC=GCGGC=GGGGGGGGGGGC5GGGCG\tAS:i:0\tXS:i:0\n",
"Reference-Clone206\t141\t*\t0\t0\t*\t*\t0\t0\tATTGTGTGTCGGGTGGTGGCCTCCAGCGCTTTTCCTCTATCATTAAGAGCTAAGCCACTCCGTGTAGTATGGTGACACAAACTACTTAGCCCATTATCGAGCTCGCAGATCAGACTCCTGGGCGCGCATGGTGAGCGGAAACCTCGGGTG\tCCCGGGGG=GGGGGIIIIGGIGCIICGIIIIGCCIIII8ICICIIICIGGIIIIGCCG=58GGCGGGGIGG=GGGGCGCGCCG5CGCCGCCGGGGG8GC5GG==CIIGGGG=CC=CCCGGGGCGG8CGGGG5CG=GGGGCG8CGCCCGGG\tAS:i:0\tXS:i:0\n",
"Reference-Clone204\t99\tReference\t34431\t60\t150M\t=\t34701\t420\tCCACTAATGAAGTGCTTGCTAGCTAAGTGCTTTGCCCTATCTTTCAAGTGTCCTTTCATGGTCCCTAGCAAAAGCTAGCTTTCTCCTTGCATATGTCTTTCTCCCTTCTTCCTCTCTTTTAGATAGTGGGATGGAAGAGATTCCTAAGCC\tCCCGGGGGGGGGCCGGIICIIICCIIGIIGIIII=CIIIIIGGIGGI8IIIIIIICIIIGCGGIGGIGGGGGGGGGC=GCGGGGGGGGGG=GGGG=GGGCICGCCCCGGCGCGGGGGG8GGCGGGCGCGGGG8CGCGC5GGGCCGGGCGG\tNM:i:11\tMD:Z:0G10A12T14G6T9G5A36C2T11C9G25\tAS:i:99\tXS:i:0\n",
"Reference-Clone204\t147\tReference\t34701\t60\t150M\t=\t34431\t-420\tTATACACAGATGGTCAATATTTCTTTGGTTTCTTTCTGTGCTTTCTAGTGCCCTTCCAGTAGTTTGGCTTAAATTCTTTTCATCTCATCAGTAATGCAAAGTCTCTGGTGTGTCTGGAATTTATTGGCTCCAGATTGAATTCATGAGTAC\t=CCCCCGCC=CGCGCCCGGGCGG5CGCGG=CGG=GG5GGGGGCCIICC=GGCGGGGG=CGCC===CGGGGCGGCGCGGGIGCGIGIGGI=G=CICGGGCGIIGIIIIIIGIGIIIIIGIIIIIIIIIIIGIGGI5IICGGGGG5GGG=CC\tNM:i:6\tMD:Z:22G44T23C44A4A6C1\tAS:i:123\tXS:i:0\n",
"Reference-Clone202\t99\tReference\t336418\t0\t150M\t=\t336689\t421\tAGCCGCGCGCCGCCGTCCACTGACGACTTGCATGCCGCTCCCGTCCCCGACGTGCCGCCGTTGTCGCACCTCGCCGCCGCCACCGGACGCCAATCCGGCTGCTGCTGGGCGCTGCTTCCGCTAGAGCCTACCTCGGCAACGATCTCCAAG\tCC5GGGG=GGGGGGIGGGIIICIIIC=GIIIIIIIGIGGIIICGIIGCIGIIGGGIGGCGGGIGCGG=GIGCGGGCCGGCCCGGGCGCGGG=GGGG58GGICGCCCGG==GCCG5GGGCCGGCGGGGGGCCGGG=CCGGCGGGCGGCCCC\tNM:i:6\tMD:Z:0C69A22G2T31C0T20\tAS:i:124\tXS:i:124\tXA:Z:Reference,+369571,150M,6;Reference,+402721,150M,6;\n",
"Reference-Clone202\t147\tReference\t336689\t0\t150M\t=\t336418\t-421\tACAATCTAGGCAGTGCAACAGCCACTGGTGGAGAAACCATCTTTCGTCGGTCGGGCTAATTCCACAAGAGTCCCGGTCGCAACCAAAACCGGGACTAAAGATCAACTTTAGTCCCGGTTAAAAAGGGTAACGGGCATATTTGATCTACAC\tCGG=G=GGGCGCCCCCG=CGC8CCGGGCGCGGGGCCGGG=8GCICC=CGCC8CGCG=GGCG5CCGGCGG=G=CIGCCGIG=GGGGIIIGGIG8GIGCGGGI5GCIIGGIGCICIIIGGI5IIIIGI5IIIGIGGIGCGGGGGGGGGGCCC\tNM:i:6\tMD:Z:7T11T47T9T5A20T45\tAS:i:120\tXS:i:120\tXA:Z:Reference,-369842,150M,6;Reference,-402992,150M,8;\n"
]
}
],
"source": [
"tail Clone10.sam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convert sam into bam `samtools view` \n"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[samopen] SAM header is present: 1 sequences.\n"
]
}
],
"source": [
"samtools view -@4 -bh -S -o Clone$i.bam Clone$i.sam "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Check that the bam file have been created \n",
"\n",
"* Have a look at the filesize of the sam and bam files.\n",
"* Remove the sam file "
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 196M\n",
"-rw-r--r-- 1 jovyan users 38M Jun 21 21:13 Clone10.bam\n",
"-rw-r--r-- 1 jovyan users 18M Jun 21 21:03 Clone10.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 116M Jun 21 21:13 Clone10.sam\n",
"-rw-r--r-- 1 jovyan users 25M Jun 21 21:03 Clone10.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 381 Jun 21 21:03 Clone10.sorted.flagstat\n"
]
}
],
"source": [
"ls -lh\n",
"rm Clone$i.sam"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"## Sorting raw bam `samtools sort` \n",
"\n",
"* Generate the bam file ordered\n",
"* Check that the new bam file have been created\n",
"* Remove the bam file previously created (Clone$i.mappedpaired.bam)"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [],
"source": [
"samtools sort -@4 Clone$i.bam Clone$i.sorted \n",
"rm Clone$i.bam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculate stats from mapping `samtools flagstat` "
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [],
"source": [
"samtools flagstat Clone$i.sorted.bam >Clone$i.sorted.flagstat"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Display the content of the flagstat file \n"
]
},
{
"cell_type": "code",
"execution_count": 91,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"293035 + 0 in total (QC-passed reads + QC-failed reads)\n",
"0 + 0 duplicates\n",
"225163 + 0 mapped (76.84%:-nan%)\n",
"293035 + 0 paired in sequencing\n",
"146506 + 0 read1\n",
"146529 + 0 read2\n",
"213817 + 0 properly paired (72.97%:-nan%)\n",
"223617 + 0 with itself and mate mapped\n",
"1546 + 0 singletons (0.53%:-nan%)\n",
"0 + 0 with mate mapped to a different chr\n",
"0 + 0 with mate mapped to a different chr (mapQ>=5)\n"
]
}
],
"source": [
"cat Clone10.sorted.flagstat"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generate a bam file that contains only the reads correctly paired mapped `samtools view` \n",
"\n",
"https://broadinstitute.github.io/picard/explain-flags.html"
]
},
{
"cell_type": "code",
"execution_count": 92,
"metadata": {},
"outputs": [],
"source": [
"samtools view -bh -@4 -f 0x02 -o Clone$i.sorted.mappedpaired.bam Clone$i.sorted.bam "
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"## Indexing bam files `samtools index` \n",
"\n",
"In this course we will work with two bam files :\n",
" \n",
"`Clone$i.sorted.mappedpaired.bam` will be used to SNP calling\n",
"\n",
"`Clone$i.sorted.bam` will be used to SV detection\n",
" \n",
"#### Why ?\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 93,
"metadata": {},
"outputs": [],
"source": [
"samtools index Clone$i.sorted.mappedpaired.bam\n",
"samtools index Clone$i.sorted.bam"
]
},
{
"cell_type": "code",
"execution_count": 94,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 60376\n",
"-rw-r--r-- 1 jovyan users 18213243 Jun 21 21:03 Clone10.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 25692125 Jun 21 21:13 Clone10.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 381 Jun 21 21:14 Clone10.sorted.flagstat\n",
"-rw-r--r-- 1 jovyan users 17901802 Jun 21 21:14 Clone10.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2872 Jun 21 21:14 Clone10.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 2872 Jun 21 21:14 Clone10.sorted.bam.bai\n"
]
}
],
"source": [
"ls -lrt"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"## => Let's map with data from all clones using a loop for mapping, with a single folder per sample "
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone1\n",
"\n",
">>>> Declare variables1\n",
"\n",
">>>> Mapping Clone1\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 39996000, nseq: 266640\n",
"\t[0000][ M::kt_pipeline] read 266640 sequences (39996000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 266640\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 110050, 0, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (398.35, 24.91)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 266640 reads in 25.358 CPU sec, 3.356 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.083934 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 4.53 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.84, (0.84, 0.84)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 4.51\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 3.36, (3.36, 3.36)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 0.53, (0.53, 0.53)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 2.80, (2.80, 2.80)\n",
"\t\tSMEM compute avg: 0.91, (0.94, 0.87)\n",
"\t\tSAL compute avg: 0.17, (0.21, 0.14)\n",
"\t\tBSW time, avg: 1.64, (1.66, 1.63)\n",
"\n",
"\tTotal allocs: 376853 = 376853 out total requests: 410045, Rate: 0.92\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone1\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 1\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 1\n",
"\n",
">>>> Extract only correctly mapped 1\n",
"\n",
">>>> index mappedpaired bam 1\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone2\n",
"\n",
">>>> Declare variables2\n",
"\n",
">>>> Mapping Clone2\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 39996000, nseq: 266640\n",
"\t[0000][ M::kt_pipeline] read 266640 sequences (39996000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 266640\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 110112, 0, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (398.34, 24.90)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 266640 reads in 34.260 CPU sec, 4.477 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.134329 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 5.64 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.84, (0.84, 0.84)\n",
"\tWriting IO time (SAM) avg: 0.29, (0.29, 0.29)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 5.62\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 4.48, (4.48, 4.48)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 0.97, (0.97, 0.97)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.48, (3.48, 3.48)\n",
"\t\tSMEM compute avg: 1.07, (1.09, 1.05)\n",
"\t\tSAL compute avg: 0.17, (0.20, 0.15)\n",
"\t\tBSW time, avg: 2.16, (2.19, 2.14)\n",
"\n",
"\tTotal allocs: 381886 = 381886 out total requests: 412355, Rate: 0.93\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone2\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 2\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 2\n",
"\n",
">>>> Extract only correctly mapped 2\n",
"\n",
">>>> index mappedpaired bam 2\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone3\n",
"\n",
">>>> Declare variables3\n",
"\n",
">>>> Mapping Clone3\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 39996000, nseq: 266640\n",
"\t[0000][ M::kt_pipeline] read 266640 sequences (39996000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 266640\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 110391, 0, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (397.92, 24.98)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 266640 reads in 44.563 CPU sec, 5.753 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2294.989816 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.81 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.77, (0.77, 0.77)\n",
"\tWriting IO time (SAM) avg: 0.26, (0.26, 0.26)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.01, (0.01, 0.01)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.79\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.75, (5.75, 5.75)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.91, (1.91, 1.91)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.82, (3.82, 3.82)\n",
"\t\tSMEM compute avg: 1.24, (1.26, 1.21)\n",
"\t\tSAL compute avg: 0.16, (0.20, 0.13)\n",
"\t\tBSW time, avg: 2.35, (2.36, 2.34)\n",
"\n",
"\tTotal allocs: 371807 = 371807 out total requests: 412926, Rate: 0.90\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone3\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 3\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 3\n",
"\n",
">>>> Extract only correctly mapped 3\n",
"\n",
">>>> index mappedpaired bam 3\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone4\n",
"\n",
">>>> Declare variables4\n",
"\n",
">>>> Mapping Clone4\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 39996000, nseq: 266640\n",
"\t[0000][ M::kt_pipeline] read 266640 sequences (39996000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 266640\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 97107, 4, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (377, 395, 412)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (307, 482)\n",
"[M::mem_pestat] mean and std.dev: (394.20, 26.57)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (272, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 266640 reads in 39.727 CPU sec, 5.065 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.097380 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.19 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.83, (0.83, 0.83)\n",
"\tWriting IO time (SAM) avg: 0.26, (0.26, 0.26)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.16\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.06, (5.06, 5.06)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 2.22, (2.22, 2.22)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 2.82, (2.82, 2.82)\n",
"\t\tSMEM compute avg: 1.20, (1.22, 1.17)\n",
"\t\tSAL compute avg: 0.08, (0.11, 0.05)\n",
"\t\tBSW time, avg: 1.49, (1.50, 1.48)\n",
"\n",
"\tTotal allocs: 252207 = 252207 out total requests: 391484, Rate: 0.64\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone4\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 4\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 4\n",
"\n",
">>>> Extract only correctly mapped 4\n",
"\n",
">>>> index mappedpaired bam 4\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone5\n",
"\n",
">>>> Declare variables5\n",
"\n",
">>>> Mapping Clone5\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 44856000, nseq: 299040\n",
"\t[0000][ M::kt_pipeline] read 299040 sequences (44856000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 299040\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 110039, 0, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (397.85, 25.05)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 299040 reads in 47.694 CPU sec, 6.134 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.080025 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 7.42 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.93, (0.93, 0.93)\n",
"\tWriting IO time (SAM) avg: 0.32, (0.32, 0.32)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 7.40\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 6.13, (6.13, 6.13)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 2.00, (2.00, 2.00)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 4.11, (4.11, 4.11)\n",
"\t\tSMEM compute avg: 1.40, (1.42, 1.37)\n",
"\t\tSAL compute avg: 0.14, (0.17, 0.11)\n",
"\t\tBSW time, avg: 2.49, (2.52, 2.47)\n",
"\n",
"\tTotal allocs: 371962 = 371962 out total requests: 410535, Rate: 0.91\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone5\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 5\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 5\n",
"\n",
">>>> Extract only correctly mapped 5\n",
"\n",
">>>> index mappedpaired bam 5\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone6\n",
"\n",
">>>> Declare variables6\n",
"\n",
">>>> Mapping Clone6\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 46608000, nseq: 310720\n",
"\t[0000][ M::kt_pipeline] read 310720 sequences (46608000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 310720\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 110116, 3, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (397.83, 25.10)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 310720 reads in 47.255 CPU sec, 6.051 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.144806 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 7.34 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.95, (0.95, 0.95)\n",
"\tWriting IO time (SAM) avg: 0.31, (0.31, 0.31)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 7.31\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 6.05, (6.05, 6.05)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 2.01, (2.01, 2.01)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 4.02, (4.02, 4.02)\n",
"\t\tSMEM compute avg: 1.40, (1.44, 1.37)\n",
"\t\tSAL compute avg: 0.13, (0.15, 0.09)\n",
"\t\tBSW time, avg: 2.43, (2.44, 2.41)\n",
"\n",
"\tTotal allocs: 371631 = 371631 out total requests: 410287, Rate: 0.91\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone6\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 6\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 6\n",
"\n",
">>>> Extract only correctly mapped 6\n",
"\n",
">>>> index mappedpaired bam 6\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone7\n",
"\n",
">>>> Declare variables7\n",
"\n",
">>>> Mapping Clone7\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 61170000, nseq: 407800\n",
"\t[0000][ M::kt_pipeline] read 407800 sequences (61170000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 407800\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (3, 109511, 0, 4)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (397.66, 25.27)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 407800 reads in 53.446 CPU sec, 6.822 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.141149 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 8.41 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 1.13, (1.13, 1.13)\n",
"\tWriting IO time (SAM) avg: 0.42, (0.42, 0.42)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 8.38\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 6.82, (6.82, 6.82)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 2.13, (2.13, 2.13)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 4.66, (4.66, 4.66)\n",
"\t\tSMEM compute avg: 1.94, (1.97, 1.88)\n",
"\t\tSAL compute avg: 0.13, (0.18, 0.10)\n",
"\t\tBSW time, avg: 2.52, (2.53, 2.51)\n",
"\n",
"\tTotal allocs: 372839 = 372839 out total requests: 413967, Rate: 0.90\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone7\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 7\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 7\n",
"\n",
">>>> Extract only correctly mapped 7\n",
"\n",
">>>> index mappedpaired bam 7\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone8\n",
"\n",
">>>> Declare variables8\n",
"\n",
">>>> Mapping Clone8\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 80000100, nseq: 533334\n",
"\t[0000][ M::kt_pipeline] read 533334 sequences (80000100 bp)...\n",
"Reallocating initial memory allocations!!\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 1365900, nseq: 9106\n",
"\t[0000][ M::kt_pipeline] read 9106 sequences (1365900 bp)...\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 533334\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 106914, 1, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (380, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (312, 482)\n",
"[M::mem_pestat] mean and std.dev: (397.20, 25.59)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (278, 516)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 533334 reads in 56.364 CPU sec, 7.237 real sec\n",
"[0000] 2. Calling mem_process_seqs.., task: 1\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 9106\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 1883, 0, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (380, 396, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (310, 485)\n",
"[M::mem_pestat] mean and std.dev: (397.05, 25.73)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (275, 520)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 9106 reads in 1.338 CPU sec, 0.186 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.090493 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 9.49 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 1.69, (1.69, 1.69)\n",
"\tWriting IO time (SAM) avg: 0.53, (0.53, 0.53)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 9.47\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 7.42, (7.42, 7.42)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 2.10, (2.10, 2.10)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 5.28, (5.28, 5.28)\n",
"\t\tSMEM compute avg: 2.46, (2.53, 2.40)\n",
"\t\tSAL compute avg: 0.16, (0.21, 0.09)\n",
"\t\tBSW time, avg: 2.54, (2.56, 2.52)\n",
"\n",
"\tTotal allocs: 372700 = 372700 out total requests: 416603, Rate: 0.89\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone8\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 8\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 8\n",
"\n",
">>>> Extract only correctly mapped 8\n",
"\n",
">>>> index mappedpaired bam 8\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone9\n",
"\n",
">>>> Declare variables9\n",
"\n",
">>>> Mapping Clone9\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 47190000, nseq: 314600\n",
"\t[0000][ M::kt_pipeline] read 314600 sequences (47190000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 314600\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 103685, 0, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (381, 398, 415)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (313, 483)\n",
"[M::mem_pestat] mean and std.dev: (397.58, 25.21)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (279, 517)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 314600 reads in 46.927 CPU sec, 5.986 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.118733 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 7.37 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 1.01, (1.01, 1.01)\n",
"\tWriting IO time (SAM) avg: 0.35, (0.35, 0.35)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 7.35\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.99, (5.99, 5.99)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.94, (1.94, 1.94)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 4.02, (4.02, 4.02)\n",
"\t\tSMEM compute avg: 1.51, (1.53, 1.48)\n",
"\t\tSAL compute avg: 0.13, (0.16, 0.11)\n",
"\t\tBSW time, avg: 2.31, (2.32, 2.30)\n",
"\n",
"\tTotal allocs: 351480 = 351480 out total requests: 392293, Rate: 0.90\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone9\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 9\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 9\n",
"\n",
">>>> Extract only correctly mapped 9\n",
"\n",
">>>> index mappedpaired bam 9\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone10\n",
"\n",
">>>> Declare variables10\n",
"\n",
">>>> Mapping Clone10\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 44364000, nseq: 295760\n",
"\t[0000][ M::kt_pipeline] read 295760 sequences (44364000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 295760\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 90449, 1, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (380, 398, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (312, 482)\n",
"[M::mem_pestat] mean and std.dev: (397.30, 25.31)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (278, 516)\n",
"[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 295760 reads in 42.130 CPU sec, 5.458 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.023449 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.72 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.94, (0.94, 0.94)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.70\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.46, (5.46, 5.46)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.82, (1.82, 1.82)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.61, (3.61, 3.61)\n",
"\t\tSMEM compute avg: 1.45, (1.48, 1.42)\n",
"\t\tSAL compute avg: 0.15, (0.19, 0.11)\n",
"\t\tBSW time, avg: 1.94, (1.96, 1.93)\n",
"\n",
"\tTotal allocs: 294308 = 294308 out total requests: 343778, Rate: 0.86\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone10\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 10\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 10\n",
"\n",
">>>> Extract only correctly mapped 10\n",
"\n",
">>>> index mappedpaired bam 10\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone11\n",
"\n",
">>>> Declare variables11\n",
"\n",
">>>> Mapping Clone11\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 43902000, nseq: 292680\n",
"\t[0000][ M::kt_pipeline] read 292680 sequences (43902000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 292680\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 90675, 12, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (380, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (312, 482)\n",
"[M::mem_pestat] mean and std.dev: (396.90, 25.48)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (278, 516)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (4363, 4393, 4411)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (4267, 4507)\n",
"[M::mem_pestat] mean and std.dev: (4383.92, 34.47)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (4219, 4555)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 292680 reads in 36.947 CPU sec, 4.736 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2294.955279 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.09 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 1.02, (1.02, 1.02)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.06\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 4.74, (4.74, 4.74)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.60, (1.60, 1.60)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.11, (3.11, 3.11)\n",
"\t\tSMEM compute avg: 1.32, (1.35, 1.29)\n",
"\t\tSAL compute avg: 0.10, (0.13, 0.08)\n",
"\t\tBSW time, avg: 1.62, (1.63, 1.60)\n",
"\n",
"\tTotal allocs: 275902 = 275902 out total requests: 328599, Rate: 0.84\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone11\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 11\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 11\n",
"\n",
">>>> Extract only correctly mapped 11\n",
"\n",
">>>> index mappedpaired bam 11\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone12\n",
"\n",
">>>> Declare variables12\n",
"\n",
">>>> Mapping Clone12\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 39342000, nseq: 262280\n",
"\t[0000][ M::kt_pipeline] read 262280 sequences (39342000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 262280\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 73363, 12, 2)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (378, 396, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (306, 486)\n",
"[M::mem_pestat] mean and std.dev: (395.56, 26.34)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (270, 522)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (3160, 3218, 5600)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 10480)\n",
"[M::mem_pestat] mean and std.dev: (4342.33, 1987.02)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 12920)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 262280 reads in 29.131 CPU sec, 3.730 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.119711 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 4.86 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.87, (0.87, 0.87)\n",
"\tWriting IO time (SAM) avg: 0.23, (0.23, 0.23)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 4.84\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 3.73, (3.73, 3.73)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.38, (1.38, 1.38)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 2.33, (2.33, 2.33)\n",
"\t\tSMEM compute avg: 1.15, (1.16, 1.14)\n",
"\t\tSAL compute avg: 0.07, (0.08, 0.06)\n",
"\t\tBSW time, avg: 1.06, (1.06, 1.05)\n",
"\n",
"\tTotal allocs: 204976 = 204976 out total requests: 282492, Rate: 0.73\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone12\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 12\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 12\n",
"\n",
">>>> Extract only correctly mapped 12\n",
"\n",
">>>> index mappedpaired bam 12\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone13\n",
"\n",
">>>> Declare variables13\n",
"\n",
">>>> Mapping Clone13\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 43926000, nseq: 292840\n",
"\t[0000][ M::kt_pipeline] read 292840 sequences (43926000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 292840\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 95163, 104, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (380, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (312, 482)\n",
"[M::mem_pestat] mean and std.dev: (397.10, 25.39)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (278, 516)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (442, 465, 9237)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 26827)\n",
"[M::mem_pestat] mean and std.dev: (3137.87, 4080.76)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 35622)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 292840 reads in 41.003 CPU sec, 5.267 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.149275 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.49 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.89, (0.89, 0.89)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.47\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.27, (5.27, 5.27)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.71, (1.71, 1.71)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.53, (3.53, 3.53)\n",
"\t\tSMEM compute avg: 1.41, (1.45, 1.36)\n",
"\t\tSAL compute avg: 0.12, (0.16, 0.09)\n",
"\t\tBSW time, avg: 1.94, (1.95, 1.92)\n",
"\n",
"\tTotal allocs: 294499 = 294499 out total requests: 345196, Rate: 0.85\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone13\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 13\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 13\n",
"\n",
">>>> Extract only correctly mapped 13\n",
"\n",
">>>> index mappedpaired bam 13\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone14\n",
"\n",
">>>> Declare variables14\n",
"\n",
">>>> Mapping Clone14\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 44316000, nseq: 295440\n",
"\t[0000][ M::kt_pipeline] read 295440 sequences (44316000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 295440\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 92239, 180, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (379, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (309, 484)\n",
"[M::mem_pestat] mean and std.dev: (396.71, 25.84)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (274, 519)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (139, 566, 7517)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 22273)\n",
"[M::mem_pestat] mean and std.dev: (2279.89, 3149.63)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 29651)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 295440 reads in 41.770 CPU sec, 5.330 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.118232 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.59 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.93, (0.93, 0.93)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.57\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.33, (5.33, 5.33)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.76, (1.76, 1.76)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.54, (3.54, 3.54)\n",
"\t\tSMEM compute avg: 1.40, (1.43, 1.38)\n",
"\t\tSAL compute avg: 0.10, (0.12, 0.08)\n",
"\t\tBSW time, avg: 1.98, (1.99, 1.96)\n",
"\n",
"\tTotal allocs: 302954 = 302954 out total requests: 353325, Rate: 0.86\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone14\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 14\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 14\n",
"\n",
">>>> Extract only correctly mapped 14\n",
"\n",
">>>> index mappedpaired bam 14\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone15\n",
"\n",
">>>> Declare variables15\n",
"\n",
">>>> Mapping Clone15\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 47310000, nseq: 315400\n",
"\t[0000][ M::kt_pipeline] read 315400 sequences (47310000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 315400\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 90308, 278, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (379, 396, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (309, 484)\n",
"[M::mem_pestat] mean and std.dev: (396.07, 26.11)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (274, 519)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (203, 372, 443)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 923)\n",
"[M::mem_pestat] mean and std.dev: (284.08, 111.16)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 1163)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 315400 reads in 42.231 CPU sec, 5.419 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.133584 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.85 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 1.05, (1.05, 1.05)\n",
"\tWriting IO time (SAM) avg: 0.34, (0.34, 0.34)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.82\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.42, (5.42, 5.42)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.83, (1.83, 1.83)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.56, (3.56, 3.56)\n",
"\t\tSMEM compute avg: 1.47, (1.49, 1.44)\n",
"\t\tSAL compute avg: 0.11, (0.14, 0.08)\n",
"\t\tBSW time, avg: 1.91, (1.92, 1.89)\n",
"\n",
"\tTotal allocs: 308844 = 308844 out total requests: 362672, Rate: 0.85\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone15\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 15\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 15\n",
"\n",
">>>> Extract only correctly mapped 15\n",
"\n",
">>>> index mappedpaired bam 15\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone16\n",
"\n",
">>>> Declare variables16\n",
"\n",
">>>> Mapping Clone16\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 50862000, nseq: 339080\n",
"\t[0000][ M::kt_pipeline] read 339080 sequences (50862000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 339080\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 94406, 607, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (377, 396, 413)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (305, 485)\n",
"[M::mem_pestat] mean and std.dev: (395.17, 26.90)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (269, 521)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (209, 356, 1125)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2957)\n",
"[M::mem_pestat] mean and std.dev: (379.56, 374.06)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 3873)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 339080 reads in 44.555 CPU sec, 5.677 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.116203 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 7.13 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 1.08, (1.08, 1.08)\n",
"\tWriting IO time (SAM) avg: 0.34, (0.34, 0.34)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 7.11\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.68, (5.68, 5.68)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.89, (1.89, 1.89)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.75, (3.75, 3.75)\n",
"\t\tSMEM compute avg: 1.51, (1.53, 1.48)\n",
"\t\tSAL compute avg: 0.10, (0.13, 0.09)\n",
"\t\tBSW time, avg: 2.07, (2.08, 2.06)\n",
"\n",
"\tTotal allocs: 346598 = 346598 out total requests: 418124, Rate: 0.83\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone16\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 16\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 16\n",
"\n",
">>>> Extract only correctly mapped 16\n",
"\n",
">>>> index mappedpaired bam 16\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone17\n",
"\n",
">>>> Declare variables17\n",
"\n",
">>>> Mapping Clone17\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 45186000, nseq: 301240\n",
"\t[0000][ M::kt_pipeline] read 301240 sequences (45186000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 301240\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 94736, 295, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (380, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (312, 482)\n",
"[M::mem_pestat] mean and std.dev: (396.78, 25.76)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (278, 516)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (328, 414, 570)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1054)\n",
"[M::mem_pestat] mean and std.dev: (394.39, 148.04)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 1296)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 301240 reads in 40.426 CPU sec, 5.207 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.011762 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.43 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.89, (0.89, 0.89)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.41\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.21, (5.21, 5.21)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.64, (1.64, 1.64)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.54, (3.54, 3.54)\n",
"\t\tSMEM compute avg: 1.46, (1.50, 1.40)\n",
"\t\tSAL compute avg: 0.12, (0.18, 0.08)\n",
"\t\tBSW time, avg: 1.89, (1.91, 1.88)\n",
"\n",
"\tTotal allocs: 291951 = 291951 out total requests: 343993, Rate: 0.85\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone17\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 17\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 17\n",
"\n",
">>>> Extract only correctly mapped 17\n",
"\n",
">>>> index mappedpaired bam 17\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone18\n",
"\n",
">>>> Declare variables18\n",
"\n",
">>>> Mapping Clone18\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 44592000, nseq: 297280\n",
"\t[0000][ M::kt_pipeline] read 297280 sequences (44592000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 297280\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 87002, 211, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (379, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (309, 484)\n",
"[M::mem_pestat] mean and std.dev: (396.52, 25.88)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (274, 519)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (246, 299, 522)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1074)\n",
"[M::mem_pestat] mean and std.dev: (356.28, 176.51)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 1350)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 297280 reads in 40.500 CPU sec, 5.204 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.070891 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.47 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.94, (0.94, 0.94)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.45\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.20, (5.20, 5.20)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.82, (1.82, 1.82)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.35, (3.35, 3.35)\n",
"\t\tSMEM compute avg: 1.41, (1.45, 1.37)\n",
"\t\tSAL compute avg: 0.11, (0.14, 0.08)\n",
"\t\tBSW time, avg: 1.76, (1.77, 1.76)\n",
"\n",
"\tTotal allocs: 269194 = 269194 out total requests: 320377, Rate: 0.84\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone18\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 18\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 18\n",
"\n",
">>>> Extract only correctly mapped 18\n",
"\n",
">>>> index mappedpaired bam 18\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone19\n",
"\n",
">>>> Declare variables19\n",
"\n",
">>>> Mapping Clone19\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 46578000, nseq: 310520\n",
"\t[0000][ M::kt_pipeline] read 310520 sequences (46578000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 310520\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 93078, 36, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (379, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (309, 484)\n",
"[M::mem_pestat] mean and std.dev: (396.64, 25.81)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (274, 519)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (444, 465, 2335)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 6117)\n",
"[M::mem_pestat] mean and std.dev: (976.56, 855.83)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 8008)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 310520 reads in 42.235 CPU sec, 5.481 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2295.112641 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.79 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.98, (0.98, 0.98)\n",
"\tWriting IO time (SAM) avg: 0.30, (0.30, 0.30)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.77\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.48, (5.48, 5.48)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.78, (1.78, 1.78)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.67, (3.67, 3.67)\n",
"\t\tSMEM compute avg: 1.51, (1.57, 1.47)\n",
"\t\tSAL compute avg: 0.15, (0.20, 0.10)\n",
"\t\tBSW time, avg: 1.94, (1.95, 1.93)\n",
"\n",
"\tTotal allocs: 305079 = 305079 out total requests: 355724, Rate: 0.86\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone19\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 19\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 19\n",
"\n",
">>>> Extract only correctly mapped 19\n",
"\n",
">>>> index mappedpaired bam 19\n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone20\n",
"\n",
">>>> Declare variables20\n",
"\n",
">>>> Mapping Clone20\n",
"\n",
"-----------------------------\n",
"Executing in AVX2 mode!!\n",
"-----------------------------\n",
"Ref file: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"Entering FMI_search\n",
"reference seq len = 2040003\n",
"count\n",
"0,\t1\n",
"1,\t576484\n",
"2,\t1020002\n",
"3,\t1463520\n",
"4,\t2040003\n",
"\n",
"Reading other elements of the index from files /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"prefix: /home/jovyan/work/SV_DATA/REF/reference.fasta\n",
"[M::bwa_idx_load_ele] read 0 ALT contigs\n",
"Done reading Index!!\n",
"Reading reference genome..\n",
"Binary seq file = /home/jovyan/work/SV_DATA/REF/reference.fasta.0123\n",
"Reference genome size: 2040002 bp\n",
"Done readng reference genome !!\n",
"\n",
"[0000] 1: Calling process()\n",
"\n",
"Threads used (compute): 8\n",
"Info: projected #read in a task: 529811\n",
"------------------------------------------\n",
"Memory pre-allocation for chaining: 1114.7223 MB\n",
"Memory pre-allocation for BSW: 1916.9362 MB\n",
"Memory pre-allocation for BWT: 618.5134 MB\n",
"------------------------------------------\n",
"No. of pipeline threads: 2\n",
"[0000] read_chunk: 80000000, work_chunk_size: 43560000, nseq: 290400\n",
"\t[0000][ M::kt_pipeline] read 290400 sequences (43560000 bp)...\n",
"[0000] 2. Calling mem_process_seqs.., task: 0\n",
"[0000] 3. Calling kt_for - worker_bwt\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] 3. Calling kt_for - worker_aln\n",
"Inferring insert size distribution from data, l_pac: 1020001, n: 290400\n",
"[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 87298, 134, 0)\n",
"[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
"[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (379, 397, 414)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (309, 484)\n",
"[M::mem_pestat] mean and std.dev: (396.64, 25.85)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (274, 519)\n",
"[M::mem_pestat] analyzing insert size distribution for orientation RF...\n",
"[M::mem_pestat] (25, 50, 75) percentile: (134, 190, 252)\n",
"[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 488)\n",
"[M::mem_pestat] mean and std.dev: (211.16, 91.90)\n",
"[M::mem_pestat] low and high boundaries for proper pairs: (1, 606)\n",
"[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
"[M::mem_pestat] skip orientation RF\n",
"[0000] 10. Calling kt_for - worker_sam\n",
"\t[0000][ M::mem_process_seqs] Processed 290400 reads in 40.668 CPU sec, 5.209 real sec\n",
"[0000] read_chunk: 80000000, work_chunk_size: 0, nseq: 0\n",
"[0000] Computation ends..\n",
"No. of OMP threads: 8\n",
"Processor is runnig @2294.987879 MHz\n",
"Runtime profile:\n",
"\n",
"\t Time taken for main_mem function: 6.48 Sec\n",
"\n",
"\tIO times (sec) :\n",
"\tReading IO time (reads) avg: 0.92, (0.92, 0.92)\n",
"\tWriting IO time (SAM) avg: 0.32, (0.32, 0.32)\n",
"\tReading IO time (Reference Genome) avg: 0.00, (0.00, 0.00)\n",
"\tIndex read time avg: 0.02, (0.02, 0.02)\n",
"\n",
"\tOverall time (sec) (Excluding Index reading time):\n",
"\tPROCESS() (Total compute time + (read + SAM) IO time) : 6.45\n",
"\tMEM_PROCESS_SEQ() (Total compute time (Kernel + SAM)), avg: 5.21, (5.21, 5.21)\n",
"\n",
"\t SAM Processing time (sec):\n",
"\t--WORKER_SAM avg: 1.77, (1.77, 1.77)\n",
"\n",
"\tKernels' compute time (sec):\n",
"\tTotal kernel (smem+sal+bsw) time avg: 3.42, (3.42, 3.42)\n",
"\t\tSMEM compute avg: 1.34, (1.37, 1.30)\n",
"\t\tSAL compute avg: 0.10, (0.14, 0.08)\n",
"\t\tBSW time, avg: 1.91, (1.92, 1.89)\n",
"\n",
"\tTotal allocs: 299305 = 299305 out total requests: 351940, Rate: 0.85\n",
"\n",
"Important parameter settings: \n",
"\tBATCH_SIZE: 512\n",
"\tMAX_SEQ_LEN_REF: 256\n",
"\tMAX_SEQ_LEN_QER: 128\n",
"\tMAX_SEQ_LEN8: 128\n",
"\tSEEDS_PER_READ: 500\n",
"\tSIMD_WIDTH8 X: 32\n",
"\tSIMD_WIDTH16 X: 16\n",
"\tAVG_SEEDS_PER_READ: 64\n",
"\n",
">>>> convert sam to bam for Clone20\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
">>>> Sort raw bam file 20\n",
"\n",
">>>> Flagstats from all reads using by default sorted bam 20\n",
"\n",
">>>> Extract only correctly mapped 20\n",
"\n",
">>>> index mappedpaired bam 20\n"
]
}
],
"source": [
"for i in {1..20}\n",
" do\n",
" cd ~/work/MAPPING-ILL\n",
" echo -e \"\\n\\n>>>>>>>>>> Creation directory for Clone$i\"\n",
" mkdir -p dirClone$i\n",
" cd dirClone$i\n",
" \n",
" echo -e \"\\n>>>> Declare variables$i\"\n",
" REF=\"/home/jovyan/work/SV_DATA/REF/reference.fasta\"\n",
" ILL_R1=\"/home/jovyan/work/SV_DATA/SHORT_READS/Clone${i}_R1.fastq.gz\"\n",
" ILL_R2=\"/home/jovyan/work/SV_DATA/SHORT_READS/Clone${i}_R2.fastq.gz\"\n",
"\n",
" echo -e \"\\n>>>> Mapping Clone$i\\n\"\n",
" bwa-mem2 mem -M -R \"@RG\\tID:Clone${i}\\tSM:Clone${i}\" -t 8 $REF $ILL_R1 $ILL_R2 > Clone$i.sam\n",
" \n",
" echo -e \"\\n>>>> convert sam to bam for Clone$i\"\n",
" samtools view -@4 -bh -S -o Clone$i.bam Clone$i.sam \n",
" rm Clone$i.sam\n",
" \n",
" echo -e \"\\n>>>> Sort raw bam file $i\"\n",
" samtools sort -@4 Clone$i.bam Clone$i.sorted \n",
" rm Clone$i.bam\n",
"\n",
" echo -e \"\\n>>>> Flagstats from all reads using by default sorted bam $i\"\n",
" samtools flagstat Clone$i.sorted.bam >Clone$i.sorted.flagstat\n",
" \n",
" echo -e \"\\n>>>> Extract only correctly mapped $i\"\n",
" samtools view -bh -@4 -f 0x02 -o Clone$i.sorted.mappedpaired.bam Clone$i.sorted.bam \n",
" \n",
" echo -e \"\\n>>>> index mappedpaired bam $i\"\n",
" samtools index Clone$i.sorted.mappedpaired.bam\n",
" samtools index Clone$i.sorted.bam\n",
" \n",
" done"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2.2 Mapping Long reads vs a Reference `minimap2` \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similar process such as SR is done in LR. In this case mapper is minimap2."
]
},
{
"cell_type": "code",
"execution_count": 95,
"metadata": {},
"outputs": [],
"source": [
"# Declare variables\n",
"i=10\n",
"REF_DIR=\"/home/jovyan/work/SV_DATA/REF/\"\n",
"REF=\"/home/jovyan/work/SV_DATA/REF/reference.fasta\"\n",
"ONT=\"/home/jovyan/work/SV_DATA/LONG_READS/Clone${i}.fastq.gz\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Mapping with `minimap2` "
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"## => Let's map now but only WITH READS FROM ONLY ONE CLONE "
]
},
{
"cell_type": "code",
"execution_count": 101,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Creation directory for Clone10\n",
"\n",
"Clone10\n"
]
}
],
"source": [
"mkdir -p ~/work/MAPPING-ONT\n",
"cd ~/work/MAPPING-ONT\n",
"echo -e \"\\nCreation directory for Clone$i\\n\"\n",
"echo Clone$i\n",
"mkdir -p dirClone$i\n",
"cd dirClone$i"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Alignement between a reference and ONT reads ̀minimap2` "
]
},
{
"cell_type": "code",
"execution_count": 107,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Mapping Clone10 minimap2 \n",
"\n",
"[M::mm_idx_gen::0.082*1.06] collected minimizers\n",
"[M::mm_idx_gen::0.112*3.08] sorted minimizers\n",
"[M::main::0.112*3.08] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.122*2.90] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.128*2.82] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.195*7.32] mapped 11235 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone10.fastq.gz\n",
"[M::main] Real time: 9.211 sec; CPU: 67.304 sec; Peak RSS: 0.585 GB\n"
]
}
],
"source": [
"echo -e \"\\nMapping Clone$i minimap2 \\n\"\n",
"minimap2 -ax map-ont -t 12 ${REF} ${ONT} > Clone${i}_ONT.sam "
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"@SQ\tSN:Reference\tLN:1020001\n",
"@PG\tID:minimap2\tPN:minimap2\tVN:2.17-r941\tCL:minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone10.fastq.gz\n",
"86f59255-6632-405c-a329-62d9dba8a95f\t0\tReference\t495672\t60\t3157S38M1D8M1D49M1I3M1D8M1I37M2I5M2D52M1D10M2I8M1D11M1I18M1I17M1I62M1D56M1D27M2D32M3D16M1D3M3I41M1I5M1D3M4I6M1I3M1I2M1I6M1D164M4I13M1I29M1D2M1I22M1I3M2I6M2D14M1D15M1D47M1I61M1D32M3D13M2I17M1I60M1D22M1I62M1D17M2I15M1D8M1I2M1I21M1D22M1D2M1D37M1D5M2I3M1D5M3D3M2D27M1I8M1D17M1I8M1I4M1I5M1I13M2I42M1I2M1I24M1I6M1I1M1I39M1D38M3D11M1I9M2D12M1I1M1I44M1I36M1D11M1D48M2I4M2I5M2I103M1I2M2D20M2D45M1I15M1I37M1I10M1I5M1I42M2I23M1D17M7S\t*\t0\t0\tTATTTTCAAATACTAAATGATTTCAACTGAAAACGTCATCAATAACTCAAAGTTGTATTAATCATCAAGATCTATAACTTTCATTTTGGTCAGTTCTTCATCGGACAAAGTAATTTGTAAGATTGTTCCACAAGATGTACTTATCTTTTATATAGTTAATAAAAACTATAAGAGTGGTTACATTTTGTGAACAGTCTTATTAATAACTTTGTCGGATGAAGAAATGTCTAAATGGGTTATAGATCTTGCTGAGTTATACAACTACGTTCTTGATGACTTTTCAGCCGAAATCAAATACTACTGCAAAATATTGTTTGAAGTTTTGAAATTCAGAATTTTAATTGATAAAAGTCACAATAAAAAATGGCCAAAATAATAGCAGTAAGAACACAATAACATGATAGAGCATGATTTTAGAAATATTTAGAAAAAGTCATCTTGTTTAGGTTTGAATAATTTGAGTTTGAATTTCAAATTCTGACAATTTCAAACAACATTTTCAAAAACTAAACGATTTTAAGTGAAAAGTGATCAACAACAAAGTTATTATAAATGATCATTATCTATACCTTTTTGTTTTGGTAATTTTGGTATATGACCTCTGTTTCAATAATTTGAATTTAAATTTCAAATTATGACCACCTGCAAACAAAATTTACAAATACTAAACGATTTCAACTGAAAAGTCATCAACAACAAAATTGTGTAACTCATCAATATCTATAACTTTTATTTTTGTCATATCTTCATCCGATAAAGTGATATTAACGTTGTTTACAAAGTTTACAAGTCATCTTATAAACTTTGATGAATTTAAATCAGTGACAAGAATTGCTATAACGTCTCAGGTTTGGGCCGGTTTATCACGCAACCGATAAACTGTCACGGTATTATGGGCCTGTATTCTGGGCCTTTCACCCATAGTTGCATATCCGTGACGATATCTTACAACATCACGAAAATCCCCTGGCCCCGCAGCGGACGTGATAATCCACAAATGTTCTGGTAAGTGAAATTGTGCTGGTCCACAGACGGTCGTCAGTCCCGAGCGATTATTGTCCAATTGAGCCTGCTAGTGAGACGGGGCGCTGGTGCTCGTTGGGCAGACGGGACAACCTGGCTCCAGTACCCAAACTGCCCCTGCCTACAAGGCTGGACCGCTTCATAAGAGATCCCTCGAACATAGAGGACGAGCTCCGGAACAAATTCTCAGGAAGAGGCTGGGGGATCAAAGGTCCACCCATGACTAGCCCAGATTCAGGCAGTCGTTCGAGGTTGTGTTTGTCACTGCAAAGGACTAACGGCCAACAATCTTAACTAAGATCGCGTCCTCTGATCAGTGTTGAGCGGATACTGCCGATGATCTGGCCTTAGTGCCCGTTAAGACATTACAGCGGAAGTCACACCGCCAACCGACCGGCTCAGAGTCAGCTCCACTCACACCGTAAGTGCAGAAGTAGAGGATGTTCCAAGGGTTAGTAGCGCCCGTACTCGTGGTCCTCCGGGAGTTCTGTTAAAAACAAACTTTATGAGGAAGTAGAGGGGACTCAAACGGATGGCCAGCAGGTCTGGCGAACTTATTATGCGGCTTAATCGCTGCTTTGCGACGTTTTCGGCCCCACTAGCTGGGGGAGCCGGTGTCACCACCACACGCTTAATTGTGGACCCATACGAGGGGAACTCGCCGCGCTCACGTACTGGAGGTTTACTAATTTGAGTCGGTGTTACGATTTTCACTAGGTACCCGGGTAAAAGCGCCGATATATCATCATTAAGAAGTTGCACCAAACTGACAGTCAGACAGGCCCAGGCTGGCTAGTAGCTCAGATGTCGACAGACAGAGGTCCTTGCTCCCCTCGTCATGCGGGTATTACTCTTGGGGCCTGAGGCTCTCTGGATGGATTCGGCGTTTTAAGTGTCACCTTATAAACTCCCCTGTTTGCATTGAATGCGTGAACCACCTGGCATCTTAGAAGATGAACCGACGTCTAAATCTACTGAAACATTCCCTATATAGGGGTTCGTGGGACCATCCTACCCGTAGCTGAAGTAAACATTTTTGCCAGCTATCCATGGGCCCAGGAGATATGGCTGGTTTTTATCAGGAACCCTCGTGGAACTACAGCAGGACCTCCGCTTTTGCTTTGCTAACCGTTTTCAAAGAGGCCGTAACGCCAAGAGACGAACATTCGTCCCAGCCGCTTTATTCCGTGGCGGCTCGACCGATTTGTTAAATTTCCGAGAAATCCTGGATGAGGACCGGTACAGCGTTGATTTACAGCACGGTTGTTACAGGTAATACGTCGTGACAAAACCACGGAACTACCGGCTGTTATAGCATGAATCGGTATTTTTAAGGACGCTGATGCTCGTTGAGCTTTCCGTTACATATTCTCCCGCCATAGCACCTCAAAGGTTTTAGTTCATCCTCCACTTCTCAGTGCGCCTTATCGTGTCATGTCCCCACGAATCCGTGGAAATCCTGGAACTATGAGGATCAACTGCTTTACTGCCCTGAGCGACTAATGTTCACCATAAGGAACCATAAATAAAGGCAAAGCGTCGATCTCACTAGTGCAGACTGATTGCCTGGGGTTCCACAGCCCAGTACAGTCAACTGTGATGACTGCGCTGGCTTTCTGCCTATATCTCGGGATCTCACTGGAGTACCCAGTTAGGTCCGCGTTTGGACCATCACCCCAACTAGGGGCCTGGCGACACACAAACGATTTTACAGACAGCGGAGGATACCTCGGGAAACAGCGCCAGTCTACCGGCCGTGGAGGTGATCGACGATATCTCTGCCTCTCCGAAGCAATCATCGTATTCTTCTCAATAAAGCCTGGAACGCTCAGCGCTTCCGACCCATCTCGGAGGGCACCCTTGCCAGGCCGTTTTACCCTGCTACCCCCGCCGCTAGCATTTATACTCATATATTTTGCGATCCGCACCGCTCCCAGTGGCATGTGATACTATTGGTAATAGAAAATTGTCAAGATAAGCTTACTCAGTCCTAACACCCCAACAGGGTCGAACACATCATTACGCTATACTCGACAGGAGACCGTTAGCACACTTCTTCCGCTTTGGGAGCAGACCCTTTGCAATAGCTTCAACATACACTTCCGGGTGTTACTAATGTCCGACCCGATCAGATTTTCGCCAAATTTCCTTACACATTAAATAGTTTCACATGTGATAATCTATATTATTAGGCCATAACTAACCATGCATGCAGATACCCTGTTAGCCACAGACCCTAATCAGCTAATACCACGTAGTGTCTATACAAATTACGGTGTTAATGACTGAAACCTGACAGGGAAATCTCGCCTATGTATACATGGATCTTGAACATTCTAGGGCAACTGAATTAATATCTAAGGCTGAACAAAGGTATATGTATATTGAGTACCTTGTAAAATAAAAGTACTATAACTATCTTTCTAGATAGCTGATGCATGCTAAGATATGTATTGAAATCTCTAAGAAAATTTAGCACAGGTTGTGAGTAAATTCTCCTTTGCTAGCTAGTTTTTCTTTAGAGAGGTTAAGTCGTAATTCACTAGTTGATATTAATAATAGTTTATTTACTACAACTGTTAAGAAAGGAGCAAGGGGACAGTCCAAACCACGAACTAGCCAACAAATGAGGTGATGTTAGAAGGACAATAATTAACAAGGGTCAAGATGAGTAAAATGGTTGCTAAGGTGAAGTTTTAGCCAGACCGTTAATATGTTAATTGAACATAATAGCCTACCTCGGATAAATCTCAGCCTCGATTATTAACTGACATGTAAAGTGGTCGATACTGATCAATTGCGTACAAGTGTCCGTTTGGATACCATCTACCATTGCTCAGAGTCCTCATTATCTAACAGATATTGCATCTCCCTCGCTATATTACAGCATATCAGCTCAGTTTCTAAATATTTAACAGTCTGGTGGACTTTTAAACATGTTTATTGATCGTTTTATTGAAAAATTTAATTAATTATTAATTATTTTCCTCTCATTTGATTCATTAATTAAATATATACACTTATATATATACATATATATATCTATAGTTTTAAATATTTCATAAATTTTATTAATTTGACAGATGGTCAAATATGTGAAAGGACAACAGTCAATCATCTATTAAAAAACTGAAGGGAGGGATCAGTATAATATTAGCGCCATTTTAGAGCCAGCTCGTCTATAAATACAGCAAATTGCATTGCCATAGTCATCCACCTCTACTTTTACTGCTAGCTAGCAGCAGCTCGATCGATATAGCTATACCATATATGGAGGCGTTTCAAGGAAACGACCACTGCCTGCCAGTGCTTGCTGTGAGTCGGTAACAACCATTGTTTTGATGTAACAACGTCGGCGGCCAGGGGGCACGGCACAGGACTTCGTGGACCTGCACAACGCGGTGGGGACAGGGATCGGCGAGGGTGACGTGGGACGACACCGTGGTGGCGGTGCGCGGGTCCACCGCGGCGCAGTCGCCAAGCCTGAATCACCAGCCCCGTAAGCACGAAGACAACGGAACAGCCACCTACGGAGAGACCATCTGCATGATCGATCATTTCCCGCCGGCGGCAACGACACGATCGTCGATCCGCCGGCCGCCGCGGCCGCGGGCGCTCAGGCGGCGGAGGGCAGTGGTTCGACCCCGACACCAGCGGCTTCTCGGCGCAGGGGGACTCGTTGCGACCACCACGCAGCTGATCGCTGGAACGCCACCACGGCCATCGCCTACGCCGAGGTCGTCTGCGACCAGCGACGCCCGGGTCTTCGTCATCTGCAACTACTACCTCCGGGCAAATCCCATTCCAGAGCCCCTACTAGCTAGGTTGCTTTGAAACCACAAGTCAATATTAATGATTAATCTATATATATATATATATCTATATATATATATATATATATATCTATAGCTACGTTATATACTATAATCGCATGCGATCATGCATGCCTGCATATACTGTTCAGTATACATAAATACCCTATATGCTCTACGTACATACTACTAGTATGTACAACGCATACTATATGTACATTGCCTACGCACGTTCAATCTAGAACGATATCAGCATGCATGTGATTGATGGAGATAAAAGTAGATCATCAATTCACATCATCTGTTGGCTAGTACCTAGATATATGCATCATCAACATGCAGGTGTACTGTACAACGGTTTATATATCCTAGTAGTGGTGC\t&388@=825368::<>+$'&'10401465=9:2-.+.)'3((('(%%/)*&-&(%)+()+1172.7$)&+(3*.*238;=>:9;<=9&'+3*,4134522+944:5>8+)'')&./()*.+)/.,68%823*1+1+)(*137>=>C;1/++./8894969.11651/134:;?<6875/5;:788;:==978:BD5464*8&)33*0*+04479:,(--&&,,.578777.-,)767@=;245-&'*'+(-)')'*//-79834552.865764393<;<6A5A?9>?DC<.7<:82.(,0-0-++%147*25=>?7673<=@A;75=89>:<>66<;*2.010%,()+++<2>;?9:)*59?@G@6CFMMA<297/.-,8874*-+%%+)'*%$$('7%'-345C6/923$',/0461&&.)(&)-5888+&&1245331%7895))=89?>;7'')&)*(*(+'+12.=?@:960/0/34,()+,.,3879:=4;;99<:9:4-32328;490112*0.//%)+++%',-241+((;>8>9944,?=@@*%:/*6++-./020.-)*$($%/174:)-547888:;<;78>DDGA:BAA?:'(2$(+36>;=144824-533*%+)$)$(1.(+)*57754152-&*++'($&$-1+049949;@2789763/.6/4565940'$#,-+(%)%*))*/45;89-%-,;04)47:8.&%%)..233-++'%##%&'&(*-279-;:73@8++,2424/)))+-.++-2542../0::??0?8979-%%1*+,,+.5014-04/.+-,+)010143./.-:%$&--5*11;<,+,2;;=64+3')*-.%$%%(&'(((1$$$&$$$113233215,+,32-./(;):;5)/04+))*0%$%&##%)#'%%3*&'&%$##%%.*+''9$)%%'(+544;>1/6665.2''(2.'%$*&%/'/4(+((''%%%%-(()-%#,-32,)).6%.)$)%111ACA?>=>=>3/02048>?A=94;;=C@@:;;.:*+)96564//4'(11250.,.,(()%#$($99/A23-.257<>@?;C7GGDB9*;>@@:2;@83/440(*0226.345%-/-.)*%)*(**$/0/*5-((/+.3?71'*.)*)%.'258)2;0.616:;6()%#(''%#$$'$%%$&+)),))##+0.'&0&$'34968/.'-,#%0),,0(1)');78<=:3.0,'-()6-%)0/++)07443(()(*-.*$$$$'+-.0/51655<558<92-3&(887;=>5;:--%&%(+606=C9<6=7C;3<96/.+--%%&&&')##'-$32:3.89$91488=5+541,-;<=77)-&&&&&%#+$33+,333*/'&$%1)3:54-+682/458776%($%$#)49>>>59=11122B@ACEBA;;'%&67=>@A/=6=>=<;-*,*+)(*)/$')*(-$$##%%'%%2463362..'''$%$$%(&&&''*+4488651/-+42044:&)+2))%&'$-,.(&$&$#%%+%%&'()-&.///1;88542*)$$)%/)6&'&)&&*%##$'##%%%&'&&,%)04=6:91/-418400***.,//1/01-))+,.38+++*378-2/&$&10/,+'3./$(()1855+/(0&,3-,*(($,+.-0469:84-674&)%$$&&+'##$)))887AEEEAA@96;:=9671*'%%&-&'(*$$24/12-8875762,*'%%03-/-+0*+'')()0(+$#$&(//(*0**12147/($)*#$$(&%%(*&''68>>>)'672/00009*%$%$01/*0/-0-&&%$'04'((.--+'&8'542-350/$$('/.,*&$#&%$$32.'((*-47CBA4*225@38=,,13:@;>64=A;5.,3110(45//,3(2)%*&%%&'((5.*(%%8)2)45'$(*')&/.63253(,-41.,+(+-+AA7++--47?@A+8/.022222:464/'&$'8<78,39/*'($&%%)$)**&'&+0100435%00,*+/3189961467/1578;;;+//02172034835.-(+(#&*+.23610)($='044552218=78;(('*,+.'*+*%%&)0&??=?AAA@-+,-)*&&$&%%''&)%#',)'(+**%&+4-,03.)&+&%*-,./%$$+(#%$.,9,-'$*0$$#&''(&%&)*+,))(%'##$&*)*/+//(./+&&2.0%%%&%$&%*--120-&%3.2899:47142*&%..617()7;55-6*/1(($&$').)%&$$&'%&&5$%%'-+((#&')06)2$$$4592/3..173,,//0'457677;4621CEA55555.&&-**+(2'$&)***:()601&$$40391&'*)$$%&$(111111++.&')*'&%%)(()))'+-*0101/$/')($%$'/3,+&$%&.0,6455<23377''533:C:6(()48146@=789549CB:',)0/$)0/1.0$$(01544*%%%)'11124&&,.)''(22697,(&+.78761/*#'%$+42+(-4789+,((,***.%%10.642,''*%%%'$&%'&*-2,''3//2/%&+''%$$*4<::-7=>:>399;5./8;5;@A:<82249+9;9/3/>@B@7788>>8;;<=)<0-/2(9:<>==;6333520/641-'8:5114879-.8=??A@>04-1;1B<<<=<>=5579=<4$''&&/5680-$;*.',$&-##))-35.68732-.43/+*-1++23>??>AAA50/0'$%$'(((:=@=2)-&$$$#&))''./.(('(%-$('&&$$&%)%$%&$$%'$$%$%$&%*+$&*',.+++%($$$/-(,1/1-./+'&%$'&&2,&&(79981'&*''+*0'+*&&'&&#$%&&$$/0)0/32'*'%#$$&,($&$%*+434**41,-0*&#&'&&%%'#&/2<;51/)'%*),,0//001028>?=D.744576925752/75444),*(%&%#&'&'&++(+3-8+-2%&)++++,=:=;73/21.$*&((++.559;?79>?>07+4*.*35:>BA755854/0,)0473-0-,2**+,,.,++&%'$$$.')%'('%$&$%2(599=>@B<5>BBBAEAAB)>0:49&&$&)'.'(27-&*$($%#%$&(&#%%-+)*'+,&(8466&&($%+---'**),,.2-'%$$%#$#$&((%)%%$*&$(%$%#$&(.054.-3+(161-0-+((*+0,0-+*('(&)&)&'+&$&$'%$'(&(&'-.2-,,&'()%(('$+0*532:5;;;884+%(*+,*%$#%*$%&'137501/0/41%%'$&//++*-,((),6$)%%301224$*','*)*))4,$&1...-+35)&,0,../73/%&#(##$'&%%'++4)('&')*/010372-20380,-#$(+')'('&-$''&%'(&%(()#'&,,1+-/)%%''&)(&$#).%%$,%+(478;:56.*,,&22.$%%''*,+0<=7211-;9569995.0+-+----'(%'$'3::>=05..+)..1.2/100--.$%/67860-')+007++.3/-%%,20030;888+6-4230'(,1.7A@?A:67ED:6./'-7885,*$%)**-34.$$$6''(34461))$$$%3/1+&%)$%$'#$#('%&+002303;:=560$$/11,,7-233.)-%')((-.,,,,/75369;;<:5))-*6:A;A5;305)*-342.-)+&+(+''(,,,+,.76'$($%%)$%($&((.-./3330+$)*&%&(((&%$&)'',,-'+*,1,,*,))))*+''65(&)$++64,%3+'22+3+3))))'.-.%&%&8;=;>;55265'%,(4697((%&($%%'*)''(01.3+1++++--.&&%%$*+&-+138((//9&8978/.('0,$$''>?6123/+)+39;740))2568965:77.4A0&&(&&%)*%($%'(&)#0$3363593'(%55;8941-101*+)6,0/&&'5&&6$$$2424**((*,&')+%*(-,+&%%'#$(,),333,/-2/%22..'&$,-$%*&),,((&--.22'$&'--&&)(11+456-'',*%.(%%)%$#$',,()1104,.13,*$'$&$$$%*$%)*#%&&1)*-)0>@AG@*(.453.-',>7;=40%%,(),/)53.,,&&0../1-,*'21215/23503)%%*&&('':67--0/A>BAB<82-$$%,---&%&%&.0,8862,(,($%$'*:<<66821;24112.3+<;CHA85?BL?3(-/(1-,-*($'%&'&-.2-30;=>22./.,/%'-4/+),%.+,148;;84/.$/(''(.0)++40-8:';44+'0/.-*(/*0$00/837+.,//0**+)$($$$((+('(&('($%%$%#&))&'*-)-/0%020(2+40644321/.*%%+/0/-..3*41412672,''('+)215753.)&%,/.1467352306236847898.-'-)9;;==87350,.&$$%&&+'&&-/)##,-,/1..,&-//2255@<8;;=9<::=9<@>98<>@DF;;99;A>?>995971(..0,3,&-/-1)$)#%%+/69037234789.(,(%&&*()))'$%&(),,20+4&/'*,-)%$%&%&%'))%'$-0)-,-3//32)))//2-*11131*)()&&&%-*00-(%$$%&&()'&&-+0,235;3/4951,-26:,(%'()$$*)-&$$$&$%&$$\tNM:i:294\tms:i:2336\tAS:i:2336\tnn:i:0\ttp:A:P\tcm:i:48\ts1:i:502\ts2:i:0\tde:f:0.1255\tSA:Z:Reference,495158,+,446S515M2I4253S,60,44;Reference,480912,+,446M7D4770S,60,38;\trl:i:30\n"
]
}
],
"source": [
"head -n3 Clone${i}_ONT.sam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convert sam to bam "
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n"
]
}
],
"source": [
"echo -e \"\\nConvert samtobam \\n\"\n",
"samtools view -@8 -bh -S -o Clone${i}_ONT.bam Clone${i}_ONT.sam\n",
"rm Clone${i}_ONT.sam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Sort and index RAW bam "
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Sort and index bam \n",
"\n"
]
}
],
"source": [
"echo -e \"\\nSort and index raw bam \\n\"\n",
"samtools sort -@8 Clone${i}_ONT.bam Clone${i}_ONT.sorted "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Calculate stats from mapping "
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Calculate stats from mapping\n",
"\n"
]
}
],
"source": [
"echo -e \"\\nCalculate stats from mapping\\n\"\n",
"samtools flagstat Clone${i}_ONT.sorted.bam >Clone${i}_ONT.sorted.flagstats"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Display the content of the flagstat file"
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"16020 + 0 in total (QC-passed reads + QC-failed reads)\n",
"0 + 0 duplicates\n",
"14066 + 0 mapped (87.80%:-nan%)\n",
"0 + 0 paired in sequencing\n",
"0 + 0 read1\n",
"0 + 0 read2\n",
"0 + 0 properly paired (-nan%:-nan%)\n",
"0 + 0 with itself and mate mapped\n",
"0 + 0 singletons (-nan%:-nan%)\n",
"0 + 0 with mate mapped to a different chr\n"
]
}
],
"source": [
"head Clone10_ONT.sorted.flagstats"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Filter raw bam "
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Filter raw bam \n",
"\n"
]
}
],
"source": [
"echo -e \"\\nFilter raw bam \\n\"\n",
"samtools view -@8 -bh -F 0x904 -o Clone${i}_ONT.sorted.correctlymapped.bam Clone${i}_ONT.sorted.bam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index RAW and FILTERED bam "
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {},
"outputs": [],
"source": [
"echo -e \"\\index raw and filtered bam files \\n\"\n",
"samtools index Clone${i}_ONT.sorted.bam\n",
"samtools index Clone${i}_ONT.sorted.correctlymapped.bam"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## => Let's map with data from all clones using a loop for mapping, with a single folder per sample and ONT reads "
]
},
{
"cell_type": "code",
"execution_count": 129,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
">>>>>>>>>> Creation directory for Clone1\n",
"\n",
">>>> Mapping Clone1 minimap2\n",
"\n",
"[M::mm_idx_gen::0.085*1.06] collected minimizers\n",
"[M::mm_idx_gen::0.114*3.01] sorted minimizers\n",
"[M::main::0.114*3.01] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.122*2.88] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.127*2.80] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::8.402*7.02] mapped 10241 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone1.fastq.gz\n",
"[M::main] Real time: 8.416 sec; CPU: 58.961 sec; Peak RSS: 0.485 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone2\n",
"\n",
">>>> Mapping Clone2 minimap2\n",
"\n",
"[M::mm_idx_gen::0.074*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.099*3.00] sorted minimizers\n",
"[M::main::0.099*3.00] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.106*2.86] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.111*2.78] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::8.203*6.85] mapped 10241 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone2.fastq.gz\n",
"[M::main] Real time: 8.219 sec; CPU: 56.237 sec; Peak RSS: 0.478 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone3\n",
"\n",
">>>> Mapping Clone3 minimap2\n",
"\n",
"[M::mm_idx_gen::0.079*1.05] collected minimizers\n",
"[M::mm_idx_gen::0.107*3.07] sorted minimizers\n",
"[M::main::0.107*3.07] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.115*2.92] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.121*2.83] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::8.391*7.02] mapped 10242 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone3.fastq.gz\n",
"[M::main] Real time: 8.403 sec; CPU: 58.952 sec; Peak RSS: 0.517 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone4\n",
"\n",
">>>> Mapping Clone4 minimap2\n",
"\n",
"[M::mm_idx_gen::0.086*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.111*2.72] sorted minimizers\n",
"[M::main::0.111*2.72] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.119*2.61] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.123*2.55] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.309*7.38] mapped 10243 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone4.fastq.gz\n",
"[M::main] Real time: 9.322 sec; CPU: 68.688 sec; Peak RSS: 0.490 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone5\n",
"\n",
">>>> Mapping Clone5 minimap2\n",
"\n",
"[M::mm_idx_gen::0.081*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.109*3.09] sorted minimizers\n",
"[M::main::0.109*3.08] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.121*2.88] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.126*2.80] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::8.851*7.11] mapped 11343 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone5.fastq.gz\n",
"[M::main] Real time: 8.865 sec; CPU: 62.933 sec; Peak RSS: 0.619 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone6\n",
"\n",
">>>> Mapping Clone6 minimap2\n",
"\n",
"[M::mm_idx_gen::0.081*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.110*3.01] sorted minimizers\n",
"[M::main::0.110*3.01] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.119*2.86] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.124*2.79] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.908*6.71] mapped 11746 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone6.fastq.gz\n",
"[M::main] Real time: 9.920 sec; CPU: 66.467 sec; Peak RSS: 0.599 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone7\n",
"\n",
">>>> Mapping Clone7 minimap2\n",
"\n",
"[M::mm_idx_gen::0.077*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.101*2.94] sorted minimizers\n",
"[M::main::0.101*2.93] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.110*2.79] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.115*2.71] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::11.481*6.83] mapped 15055 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone7.fastq.gz\n",
"[M::main] Real time: 11.495 sec; CPU: 78.415 sec; Peak RSS: 0.736 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone8\n",
"\n",
">>>> Mapping Clone8 minimap2\n",
"\n",
"[M::mm_idx_gen::0.086*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.107*2.52] sorted minimizers\n",
"[M::main::0.107*2.52] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.115*2.41] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.121*2.35] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::15.074*6.61] mapped 19644 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone8.fastq.gz\n",
"[M::main] Real time: 15.085 sec; CPU: 99.623 sec; Peak RSS: 0.835 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone9\n",
"\n",
">>>> Mapping Clone9 minimap2\n",
"\n",
"[M::mm_idx_gen::0.082*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.110*3.00] sorted minimizers\n",
"[M::main::0.110*2.99] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.119*2.85] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.124*2.77] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::10.242*6.79] mapped 11880 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone9.fastq.gz\n",
"[M::main] Real time: 10.256 sec; CPU: 69.541 sec; Peak RSS: 0.671 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone10\n",
"\n",
">>>> Mapping Clone10 minimap2\n",
"\n",
"[M::mm_idx_gen::0.078*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.101*2.80] sorted minimizers\n",
"[M::main::0.101*2.80] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.109*2.67] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.114*2.60] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.203*7.26] mapped 11235 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone10.fastq.gz\n",
"[M::main] Real time: 9.234 sec; CPU: 66.815 sec; Peak RSS: 0.582 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone11\n",
"\n",
">>>> Mapping Clone11 minimap2\n",
"\n",
"[M::mm_idx_gen::0.076*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.101*2.87] sorted minimizers\n",
"[M::main::0.101*2.86] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.110*2.71] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.115*2.63] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.621*7.17] mapped 11131 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone11.fastq.gz\n",
"[M::main] Real time: 9.634 sec; CPU: 68.965 sec; Peak RSS: 0.632 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone12\n",
"\n",
">>>> Mapping Clone12 minimap2\n",
"\n",
"[M::mm_idx_gen::0.078*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.103*2.89] sorted minimizers\n",
"[M::main::0.103*2.89] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.112*2.75] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.117*2.67] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.774*7.52] mapped 10092 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone12.fastq.gz\n",
"[M::main] Real time: 9.814 sec; CPU: 73.558 sec; Peak RSS: 0.717 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone13\n",
"\n",
">>>> Mapping Clone13 minimap2\n",
"\n",
"[M::mm_idx_gen::0.083*1.03] collected minimizers\n",
"[M::mm_idx_gen::0.112*3.01] sorted minimizers\n",
"[M::main::0.112*3.01] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.121*2.86] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.127*2.78] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::9.999*7.61] mapped 11136 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone13.fastq.gz\n",
"[M::main] Real time: 10.009 sec; CPU: 76.083 sec; Peak RSS: 1.013 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone14\n",
"\n",
">>>> Mapping Clone14 minimap2\n",
"\n",
"[M::mm_idx_gen::0.079*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.104*2.89] sorted minimizers\n",
"[M::main::0.104*2.88] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.112*2.75] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.117*2.67] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::10.674*7.91] mapped 11226 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone14.fastq.gz\n",
"[M::main] Real time: 10.686 sec; CPU: 84.395 sec; Peak RSS: 1.299 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone15\n",
"\n",
">>>> Mapping Clone15 minimap2\n",
"\n",
"[M::mm_idx_gen::0.081*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.109*2.96] sorted minimizers\n",
"[M::main::0.109*2.96] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.117*2.83] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.122*2.75] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::12.380*8.05] mapped 11904 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone15.fastq.gz\n",
"[M::main] Real time: 12.392 sec; CPU: 99.624 sec; Peak RSS: 1.446 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone16\n",
"\n",
">>>> Mapping Clone16 minimap2\n",
"\n",
"[M::mm_idx_gen::0.078*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.101*2.81] sorted minimizers\n",
"[M::main::0.101*2.81] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.108*2.68] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.113*2.61] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::16.222*8.92] mapped 12714 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone16.fastq.gz\n",
"[M::main] Real time: 16.239 sec; CPU: 144.766 sec; Peak RSS: 1.354 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone17\n",
"\n",
">>>> Mapping Clone17 minimap2\n",
"\n",
"[M::mm_idx_gen::0.076*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.098*2.79] sorted minimizers\n",
"[M::main::0.098*2.79] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.105*2.66] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.110*2.59] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::11.023*7.67] mapped 11418 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone17.fastq.gz\n",
"[M::main] Real time: 11.038 sec; CPU: 84.554 sec; Peak RSS: 1.072 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone18\n",
"\n",
">>>> Mapping Clone18 minimap2\n",
"\n",
"[M::mm_idx_gen::0.079*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.105*2.91] sorted minimizers\n",
"[M::main::0.105*2.91] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.114*2.77] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.118*2.70] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::10.647*7.57] mapped 11288 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone18.fastq.gz\n",
"[M::main] Real time: 10.661 sec; CPU: 80.634 sec; Peak RSS: 0.958 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone19\n",
"\n",
">>>> Mapping Clone19 minimap2\n",
"\n",
"[M::mm_idx_gen::0.079*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.107*3.06] sorted minimizers\n",
"[M::main::0.108*3.06] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.117*2.89] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.124*2.79] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::11.314*7.45] mapped 11741 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone19.fastq.gz\n",
"[M::main] Real time: 11.327 sec; CPU: 84.346 sec; Peak RSS: 0.723 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n",
"\n",
">>>>>>>>>> Creation directory for Clone20\n",
"\n",
">>>> Mapping Clone20 minimap2\n",
"\n",
"[M::mm_idx_gen::0.076*1.04] collected minimizers\n",
"[M::mm_idx_gen::0.103*3.05] sorted minimizers\n",
"[M::main::0.103*3.05] loaded/built the index for 1 target sequence(s)\n",
"[M::mm_mapopt_update::0.111*2.89] mid_occ = 10\n",
"[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1\n",
"[M::mm_idx_stat::0.117*2.81] distinct minimizers: 165344 (91.75% are singletons); average occurrences: 1.156; average spacing: 5.336\n",
"[M::worker_pipeline::10.592*7.89] mapped 10804 sequences\n",
"[M::main] Version: 2.17-r941\n",
"[M::main] CMD: minimap2 -ax map-ont -t 12 /home/jovyan/work/SV_DATA/REF/reference.fasta /home/jovyan/work/SV_DATA/LONG_READS/Clone20.fastq.gz\n",
"[M::main] Real time: 10.607 sec; CPU: 83.598 sec; Peak RSS: 0.765 GB\n",
"\n",
"Convert samtobam \n",
"\n",
"[samopen] SAM header is present: 1 sequences.\n",
"\n",
"Sort and index raw bam \n",
"\n",
"\n",
"Calculate stats from mapping\n",
"\n",
"\n",
"Filter raw bam \n",
"\n",
"\n",
"index raw and filtered bam files \n",
"\n"
]
}
],
"source": [
"for i in {1..20}\n",
" do\n",
" ONT=\"/home/jovyan/work/SV_DATA/LONG_READS/Clone${i}.fastq.gz\"\n",
" mkdir -p ~/work/MAPPING-ONT\n",
" cd ~/work/MAPPING-ONT\n",
" echo -e \"\\n>>>>>>>>>> Creation directory for Clone$i\\n\"\n",
" mkdir -p dirClone$i\n",
" cd dirClone$i\n",
" \n",
" echo -e \">>>> Mapping Clone$i minimap2\\n\"\n",
" minimap2 -ax map-ont -t 12 ${REF} ${ONT} > Clone${i}_ONT.sam \n",
" \n",
" echo -e \"\\nConvert samtobam \\n\"\n",
" samtools view -@8 -bh -S -o Clone${i}_ONT.bam Clone${i}_ONT.sam\n",
" rm Clone${i}_ONT.sam\n",
"\n",
" echo -e \"\\nSort and index raw bam \\n\"\n",
" samtools sort -@8 Clone${i}_ONT.bam Clone${i}_ONT.sorted \n",
" \n",
" echo -e \"\\nCalculate stats from mapping\\n\"\n",
" samtools flagstat Clone${i}_ONT.sorted.bam >Clone${i}_ONT.sorted.flagstats\n",
" \n",
" echo -e \"\\nFilter raw bam \\n\"\n",
" samtools view -@8 -bh -F 0x904 -o Clone${i}_ONT.sorted.correctlymapped.bam Clone${i}_ONT.sorted.bam\n",
" \n",
" echo -e \"\\nindex raw and filtered bam files \\n\"\n",
" samtools index Clone${i}_ONT.sorted.bam\n",
" samtools index Clone${i}_ONT.sorted.correctlymapped.bam\n",
"\n",
" done"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## __III - Centralize final mapping data into a single bam directory__ "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reorder BAM files into a folder only for Illumina \n"
]
},
{
"cell_type": "code",
"execution_count": 134,
"metadata": {},
"outputs": [],
"source": [
"mkdir -p ~/work/MAPPING-ILL/BAM\n",
"cd ~/work/MAPPING-ILL/\n",
"\n",
"for i in {1..20}\n",
" do\n",
" cp ~/work/MAPPING-ILL/dirClone$i/Clone$i.sorted.bam* BAM/\n",
" cp ~/work/MAPPING-ILL/dirClone$i/Clone$i.sorted.mappedpaired.bam* BAM/\n",
" done"
]
},
{
"cell_type": "code",
"execution_count": 135,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 921568\n",
"-rw-r--r-- 1 jovyan users 26179162 Jun 21 23:55 Clone10.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2856 Jun 21 23:55 Clone10.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18359524 Jun 21 23:55 Clone10.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2856 Jun 21 23:55 Clone10.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 25675858 Jun 21 23:55 Clone11.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2984 Jun 21 23:55 Clone11.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 17991875 Jun 21 23:55 Clone11.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone11.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 23173804 Jun 21 23:55 Clone12.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2888 Jun 21 23:55 Clone12.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 15578786 Jun 21 23:55 Clone12.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2888 Jun 21 23:55 Clone12.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 25694602 Jun 21 23:55 Clone13.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2984 Jun 21 23:55 Clone13.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18658785 Jun 21 23:55 Clone13.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2984 Jun 21 23:55 Clone13.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 26058896 Jun 21 23:55 Clone14.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2952 Jun 21 23:55 Clone14.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18685981 Jun 21 23:55 Clone14.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2952 Jun 21 23:55 Clone14.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 28256894 Jun 21 23:55 Clone15.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone15.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18363483 Jun 21 23:55 Clone15.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2952 Jun 21 23:55 Clone15.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 30549484 Jun 21 23:55 Clone16.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone16.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 19468148 Jun 21 23:55 Clone16.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3000 Jun 21 23:55 Clone16.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 26575750 Jun 21 23:55 Clone17.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2952 Jun 21 23:55 Clone17.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18386395 Jun 21 23:55 Clone17.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone17.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 26537701 Jun 21 23:55 Clone18.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone18.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 17341818 Jun 21 23:55 Clone18.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone18.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 27618885 Jun 21 23:55 Clone19.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone19.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18655655 Jun 21 23:55 Clone19.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2968 Jun 21 23:55 Clone19.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 21210511 Jun 21 23:55 Clone1.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone1.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 21210511 Jun 21 23:55 Clone1.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone1.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 25846408 Jun 21 23:55 Clone20.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2872 Jun 21 23:55 Clone20.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 18043345 Jun 21 23:55 Clone20.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2872 Jun 21 23:55 Clone20.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 21654551 Jun 21 23:55 Clone2.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone2.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 21654551 Jun 21 23:55 Clone2.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone2.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22228733 Jun 21 23:55 Clone3.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone3.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22228567 Jun 21 23:55 Clone3.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone3.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22982961 Jun 21 23:55 Clone4.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone4.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22861720 Jun 21 23:55 Clone4.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone4.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 25512969 Jun 21 23:55 Clone5.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone5.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22205064 Jun 21 23:55 Clone5.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone5.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 26680100 Jun 21 23:55 Clone6.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone6.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22155228 Jun 21 23:55 Clone6.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone6.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 36532337 Jun 21 23:55 Clone7.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone7.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 22109119 Jun 21 23:55 Clone7.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3096 Jun 21 23:55 Clone7.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 50205387 Jun 21 23:55 Clone8.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 3160 Jun 21 23:55 Clone8.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 21947716 Jun 21 23:55 Clone8.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 3112 Jun 21 23:55 Clone8.sorted.mappedpaired.bam.bai\n",
"-rw-r--r-- 1 jovyan users 27384190 Jun 21 23:55 Clone9.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 2944 Jun 21 23:55 Clone9.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 20973577 Jun 21 23:55 Clone9.sorted.mappedpaired.bam\n",
"-rw-r--r-- 1 jovyan users 2944 Jun 21 23:55 Clone9.sorted.mappedpaired.bam.bai\n"
]
}
],
"source": [
"ls /home/jovyan/work/MAPPING-ILL/BAM -l"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reorder BAM files into a folder only for ONT "
]
},
{
"cell_type": "code",
"execution_count": 131,
"metadata": {},
"outputs": [],
"source": [
"mkdir -p ~/work/MAPPING-ONT/BAM\n",
"cd ~/work/MAPPING-ONT/\n",
"\n",
"for i in {1..20}\n",
" do\n",
" cp ~/work/MAPPING-ONT/dirClone$i/Clone${i}_ONT.sorted.bam* BAM/\n",
" cp ~/work/MAPPING-ONT/dirClone$i/Clone${i}_ONT.sorted.correctlymapped.bam* BAM\n",
" done"
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 6831452\n",
"-rw-r--r-- 1 jovyan users 179789688 Jun 21 23:39 Clone10_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 43736 Jun 21 23:39 Clone10_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 150166466 Jun 21 23:40 Clone10_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 38048 Jun 21 23:40 Clone10_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 178441901 Jun 21 23:40 Clone11_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 42856 Jun 21 23:40 Clone11_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 149694012 Jun 21 23:40 Clone11_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 38112 Jun 21 23:40 Clone11_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 168826188 Jun 21 23:40 Clone12_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 40816 Jun 21 23:40 Clone12_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 133187269 Jun 21 23:40 Clone12_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 33808 Jun 21 23:40 Clone12_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 181824745 Jun 21 23:40 Clone13_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 45344 Jun 21 23:40 Clone13_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 150108247 Jun 21 23:40 Clone13_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 37832 Jun 21 23:40 Clone13_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 182542790 Jun 21 23:40 Clone14_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 46680 Jun 21 23:40 Clone14_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 151427368 Jun 21 23:40 Clone14_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 39696 Jun 21 23:40 Clone14_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 199762414 Jun 21 23:40 Clone15_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 49048 Jun 21 23:40 Clone15_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 158359636 Jun 21 23:40 Clone15_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 39288 Jun 21 23:40 Clone15_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 222032570 Jun 21 23:40 Clone16_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 50408 Jun 21 23:40 Clone16_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 169492588 Jun 21 23:40 Clone16_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 36920 Jun 21 23:40 Clone16_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 189139007 Jun 21 23:40 Clone17_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 46952 Jun 21 23:40 Clone17_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 152879272 Jun 21 23:40 Clone17_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 39752 Jun 21 23:40 Clone17_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 185516418 Jun 21 23:40 Clone18_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 44976 Jun 21 23:40 Clone18_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 149895506 Jun 21 23:40 Clone18_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 35800 Jun 21 23:40 Clone18_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 190320218 Jun 21 23:40 Clone19_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 48000 Jun 21 23:40 Clone19_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 156087610 Jun 21 23:40 Clone19_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 39960 Jun 21 23:40 Clone19_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 160721828 Jun 21 23:39 Clone1_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 48952 Jun 21 23:39 Clone1_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 142965695 Jun 21 23:39 Clone1_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 46296 Jun 21 23:39 Clone1_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 180122698 Jun 21 23:40 Clone20_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 47056 Jun 21 23:40 Clone20_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 148421248 Jun 21 23:40 Clone20_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 37072 Jun 21 23:40 Clone20_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 160874113 Jun 21 23:39 Clone2_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 49032 Jun 21 23:39 Clone2_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 143096277 Jun 21 23:39 Clone2_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 46440 Jun 21 23:39 Clone2_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 161238302 Jun 21 23:39 Clone3_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 49240 Jun 21 23:39 Clone3_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 143470383 Jun 21 23:39 Clone3_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 46552 Jun 21 23:39 Clone3_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 161938470 Jun 21 23:39 Clone4_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 49288 Jun 21 23:39 Clone4_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 144091926 Jun 21 23:39 Clone4_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 46456 Jun 21 23:39 Clone4_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 177571882 Jun 21 23:39 Clone5_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 51136 Jun 21 23:39 Clone5_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 157320294 Jun 21 23:39 Clone5_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 47872 Jun 21 23:39 Clone5_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 184665104 Jun 21 23:39 Clone6_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 51688 Jun 21 23:39 Clone6_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 163090339 Jun 21 23:39 Clone6_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 47448 Jun 21 23:39 Clone6_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 235081986 Jun 21 23:39 Clone7_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 58680 Jun 21 23:39 Clone7_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 205204613 Jun 21 23:39 Clone7_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 51944 Jun 21 23:39 Clone7_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 308365102 Jun 21 23:39 Clone8_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 51832 Jun 21 23:39 Clone8_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 265369719 Jun 21 23:39 Clone8_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 44504 Jun 21 23:39 Clone8_ONT.sorted.correctlymapped.bam.bai\n",
"-rw-r--r-- 1 jovyan users 187617659 Jun 21 23:39 Clone9_ONT.sorted.bam\n",
"-rw-r--r-- 1 jovyan users 49840 Jun 21 23:39 Clone9_ONT.sorted.bam.bai\n",
"-rw-r--r-- 1 jovyan users 162708272 Jun 21 23:39 Clone9_ONT.sorted.correctlymapped.bam\n",
"-rw-r--r-- 1 jovyan users 45312 Jun 21 23:39 Clone9_ONT.sorted.correctlymapped.bam.bai\n"
]
}
],
"source": [
"ls ~/work/MAPPING-ONT/BAM -l"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Bash",
"language": "bash",
"name": "bash"
},
"language_info": {
"codemirror_mode": "shell",
"file_extension": ".sh",
"mimetype": "text/x-sh",
"name": "bash"
}
},
"nbformat": 4,
"nbformat_minor": 4
}