[3dem] Fwd: cryo-EM data processing in the cloud/AWS

[Laura]

Hi Chip,

my name is Laura and I work in the Centro Nacional de Biotecnología in 
Madrid, where we develop the software Scipion.

We have explored and use cryo-EM processing in the cloud for many years 
already, we have some info here 
<https://urldefense.com/v3/__https://scipion-em.github.io/docs/release-3.0.0/docs/developer/scipion-on-the-cloud.html__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGArgIJOgrM$  > 
but I can tell you also what we have been doing.

We have some AMIs with our software in AWS but we mainly use it for 
training cause it is really expensive to launch powerful hardware types 
to process cryo-EM and we only do it if we dont have our own resources.

What we sometimes use to process real data is the European Open Science 
Cloud (EOSC) resources cause we have some agreements with (OpenStack) 
sites in EU. We launch those virtual servers or clusters using the 
Infrastructure Manager <https://urldefense.com/v3/__https://www.grycap.upv.es/im/index.php__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGArB4r-v5M$  > that 
uses ansible/docker recipes/images to set them up.

If you are interested we can discuss it or I can try to answer more 
specific questions related to instances, transfers and so on.

best regards

Laura

>
>
> ---------- Forwarded message ---------
> From: *Chip Lesburg* <chip at odysseytx.com <mailto:chip at odysseytx.com>>
> Date: Tue, Aug 30, 2022 at 5:54 PM
> Subject: [3dem] cryo-EM data processing in the cloud/AWS
> To: 3dem at ncmir.ucsd.edu <mailto:3dem at ncmir.ucsd.edu> 
> <3dem at ncmir.ucsd.edu <mailto:3dem at ncmir.ucsd.edu>>
>
>
> Hi 3DEM-ers,
>
> I wonder if anyone has experience processing cryo-EM data in the 
> cloud, specifically using AWS infrastructure? There are a few online 
> resources describing this, listed below. I am hoping to learn from 
> others’ experiences, including considerations and unexpected pitfalls.
>
> How does your choice of cloud CPU/GPU instance compare with 
> on-premises computer hardware you may have access to? How do you 
> transfer raw movies from the microscope facility to your own storage 
> area in the cloud (direct network transfer, first transfer to a disk 
> array, etc.)?
>
> https://urldefense.com/v3/__https://aws.amazon.com/blogs/hpc/tag/cryo-em/__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGArl8MCPIs$   
> <https://urldefense.com/v3/__https://aws.amazon.com/blogs/hpc/tag/cryo-em/__;!!Mih3wA!CMWEZwXyRqadyg_1XwJ40k5jk4WfLCJUR4bXgU5Fpic1vMN-Qyx2b--g4gT-aRJs-vb4U6_V7xmYwN6qAA$>
>
> https://urldefense.com/v3/__http://cryoem-tools.cloud/__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGArlZk6Czw$   
> <https://urldefense.com/v3/__http://cryoem-tools.cloud/__;!!Mih3wA!CMWEZwXyRqadyg_1XwJ40k5jk4WfLCJUR4bXgU5Fpic1vMN-Qyx2b--g4gT-aRJs-vb4U6_V7xkYBBJREQ$>
>
> https://urldefense.com/v3/__https://sites.google.com/site/emcloudprocessing/__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGAr6ODbfWY$   
> <https://urldefense.com/v3/__https://sites.google.com/site/emcloudprocessing/__;!!Mih3wA!CMWEZwXyRqadyg_1XwJ40k5jk4WfLCJUR4bXgU5Fpic1vMN-Qyx2b--g4gT-aRJs-vb4U6_V7xl13NL_gg$>
>
> https://urldefense.com/v3/__https://guide.cryosparc.com/deploy/cryosparc-on-aws__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGAr8ejxhcQ$   
> <https://urldefense.com/v3/__https://guide.cryosparc.com/deploy/cryosparc-on-aws__;!!Mih3wA!CMWEZwXyRqadyg_1XwJ40k5jk4WfLCJUR4bXgU5Fpic1vMN-Qyx2b--g4gT-aRJs-vb4U6_V7xmRZbOEcg$>
>
> https://urldefense.com/v3/__https://nramm.nysbc.org/wp-content/seminars/2017/slides/nramm_workshop_2017_cianfrocco.pdf__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGArtGnlOAw$   
> <https://urldefense.com/v3/__https://nramm.nysbc.org/wp-content/seminars/2017/slides/nramm_workshop_2017_cianfrocco.pdf__;!!Mih3wA!CMWEZwXyRqadyg_1XwJ40k5jk4WfLCJUR4bXgU5Fpic1vMN-Qyx2b--g4gT-aRJs-vb4U6_V7xnqEA2JNg$>
>
> Chip
>
> **
>
> P.S. cross-posted to the CCP-EM mailing list
>
>
> 	
> 	Charles Lesburg*
> * *Vice President, Structural Biology
> *
> *M*908-265-2064
> 301 Binney Street, Cambridge, MA 02142
> *odysseytx.com <https://urldefense.com/v3/__http://odysseytx.com__;!!Mih3wA!ARQ_pdLIz3yHTBfQnqy6MQVRQehsrMsaQFt_7kF24VFusw0xNfbVYLgNiKMNolDIhCD39fEtAt1qiGAr_Hjc5Os$  >*
>
> This message is for the named person's use only. It may contain 
> confidential, proprietary or legally privileged information. If you 
> receive this message in error, please immediately delete it and all 
> copies of it from your system, destroy any hard copies of it and 
> notify the sender. ODYSSEY THERAPEUTICS, INC. ("ODYSSEY THERAPEUTICS") 
> reserves the right to monitor all e-mail communications through its 
> networks. ODYSSEY THERAPEUTICS cannot be committed by this e-mail or 
> warrant the accuracy or validity of any information presented. Any 
> views expressed in this message are those of the individual sender.
>
> _______________________________________________
> 3dem mailing list
> 3dem at ncmir.ucsd.edu <mailto:3dem at ncmir.ucsd.edu>
> https://mail.ncmir.ucsd.edu/mailman/listinfo/3dem
>
>
> -- 
> Prof. Jose-Maria Carazo
> Biocomputing Unit, Head,
> Instruct Spanish Center, Director
> Spanish National Center for Biotechnology, CNB-CSIC
> Darwin 3, Universidad Autonoma de Madrid
> 28049 Madrid, Spain
>
>
> Cell: +34639197980
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20220831/b550849c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 153156_edd5022b-de61-48a6-bf68-1a41de35ba89.png
Type: image/png
Size: 18009 bytes
Desc: not available
URL: <http://mail.ncmir.ucsd.edu/pipermail/3dem/attachments/20220831/b550849c/attachment.png>