如何调用NCBI数据

曾嘉瑞
2023-12-01

一、NCBI各类数据储存库
1、Sequence Read Archive (SRA):raw sequence data and alignment information from
high throughput sequencing platforms including 454, illumina, SOLiD and PacBio.
此外, European Nucleotide Archive (ENA) 和 DNAnexus SRA 均有SRA数据。
2、Gene Expression Omnibus (GEO):RNA-Seq, ChIP-seq, RIP-seq, HiC-seq, methyl-seq, expression data such as microarray, SAGE or mass spectrometry data sets.
3、Database of Short Genetic Variations (dbSNP):variation data, such as single nucleotide variations, microsatellites, and small-scale insertions and deletions.
4、Database of Genomic Structural Variations (dbVar):genomic structural variation data, such as large insertions, deletions, translocations etc.
5、Database of Expressed Sequence Tags (dbEST)
6、Transcriptome Shotgun Assembly Sequence Database (TSA):transcriptome assemblies
7、Whole Genome Shotgun Submissions (WGS):incomplete genome assemblies.
二、Entrez
如何使用Entrez?NCBI提供了一个API网络接口Entrez E-utilsand 和一个命令行工具Entrez Direct。
Entrez E-utils
query URL :https://service.nih.gov?param1=value1&param2=value2
在shell中,&需要使用转义符\,或者将URL添加引号‘’
        curl -s https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?  id=AF086833.2&db=nuccore&rettype=fasta|head  
        curl -s ‘https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?id=AF086833.2&db=nuccore&rettype=fasta’|head
具体命令参考文档:https://www.ncbi.nlm.nih.gov/books/NBK25500/
Entrez Direct
具体命令参考文档:https://www.ncbi.nlm.nih.gov/books/NBK179288/
einfo
抓取描述信息,einfo -dbs 查看所有数据库类型
esearch
esearch -help 查看帮助,esearch 查找的记录可以传递给efetch下载
esearch -db … -query …  | efetch -formate … 
    esearch -db sra -query PRJNA257197 | efetch -format runinfo > info.csv (runinfo — 所有搜索的信息)
efetch
It can even produce the sequence from reverse strands(reverse complement):

三、sratoolkit
1、prefetch 可以从远程站点下载文件
            prefetch --option-file ids.list
2、fast-dump : downloads data in FASTQ format, 初始sra data 保存在 ~/ncbi/public/sra/
            fastq-dump SRR1553607     --split-files(   separate paired end reads)
下载project里多个SRA数据:
        在上一步得到的info.csv文件中,第一列为这个project里所有的SRA文件,取第一列run编号,用xargs批量下载
        cat info.csv|cut-f 1 -d’,’|grepSRR|head>ids ; cat ids |xargs-n 1 fastq-dump -X 10000 --split-files $1
3、sra-stat 
            sra-stat --xml --quick SRR1553610 生成xml报告
四、 SeqKit
            seqkit stat *.gz  查看全部fasta/q文件的简要信息
            seqkit fx2tab --name --only-id --gc *.fna 显示GC含量
            seqkit fx2tab -H -n -i -B a -B c -B ac *.fna 显示A、C、AC含量
            随机取出0.001比例的id号,并根据id从总体中提取序列:seqkit sample --proportion 0.001 duplicated-reads.fq.gz|seqkitseq --name --only-id>id.txt ;seqkit grep --pattern-file id.txt duplicated-reads.fq.gz > duplicated-reads.subset.fq.gz
            seqkit grep --pattern-file id2.txt --invert-match viral.1.1.genomic.fna.gz > clean.fa 取出list上的序列
            seqkit grep --pattern-file id2.txt viral.1.1.genomic.fna.gz|seqkitlocate–ignore-case --only-positive-strand --pattern K+ --pattern N+ 显示含K、N的数据
            seqkit rmdup --by-seq --ignore-case duplicated-reads.fq.gz 去除重复
            seqkit locate --degenerate --ignore-case --pattern-file enzymes.fa *.fna 匹配短序列如酶结合位点
            seqkit sort–by-length/full header (–by-name) or sequence content (–by-seq)
            seqkit split --by-id --id-regexp"[(.+)]" 根据ID, number of parts, size of each part, or sequence region分离
            csvtk join-H -t<(seqkit fx2tab 1.fa)<(seqkit fx2tab 2.fa)|sed’s/\t\t//’|seqkit tab2fx 合并两个文件
五、other tool
get subreads from a single ZMW : bamtools filter -in subreads.bam -out zmw.bam -tag ‘zm…’
get the consensus sequence from the subreads : # ccs  [options]  INPUT OUTPUT

 类似资料: