Main Page

From HTC-as-a-Service, KISTI
Jump to: navigation, search



High-Throughput Computing as a Service (HTCaaS)

High-Throughput Computing (HTC) consists of running many loosely-coupled tasks that are independent (there is no communication needed between them) but requires a large amount of computing power during relatively a long period of time. Middleware systems such as Condor or BOINC have successfully achieved a tremendous computing power by harnessing a large number of computing resources. However, as the number of jobs and the complexity of scientific applications increase, it becomes a challenge for the traditional middleware systems employing typically a single type of resources (e.g., clusters of workstations, desktop machines over Internet) to solve the given scientific problem within a reasonable amount of time. Also, recent emerging applications requiring millions or even billions of tasks to be processed with relatively short per task execution times have led the traditional HTC to expand into Many-Task Computing (MTC).

Therefore, to effectively support complex and demanding scientific applications, it is inevitable to harness as many computing resources as possible including Supercomputers, Grids, and even Cloud. However, it is challenging for researchers to effectively utilize available resources that are under control by independent resource providers as the number of jobs (that should be submitted at once) increase dramatically (as in parameter sweeps or N-body calculations).

We designed and implemented the HTCaaS (High-Throughput Computing as a Service) system that can hide heterogeneity and complexity of leveraging different computing resources from users, and efficiently submit a large number of jobs at once by effectively managing and exploiting of all available computing resources.

Our Design Philosophy is as followings:

  • Ease of Use: We minimize user overhead for handling a large amount of jobs & computing resources
  • Intelligent Resource Selection: HTCaaS can automatically select more responsive and effective resources and adapt to the current load by dynamically adjusting acquired resources
  • Pluggable Interface to Resources: We adopt GANGA's plugin mechanism for accessing heterogeneous computing resources without hardcoding
  • Support for Many Client Interfaces: A wide range of client interfaces are supported including a native WS-interface, Java API, and Client tools (CLI, GUI)

System Architecture & Components

HTCaaS system consists of five server-side modules (Account Manager, User Data Manager, Job Manager, Agent Manager, Monitoring Manager) and two client-side tools (Command-Line Interface and Graphic User Interface).


HTCaaS Architecture


A job in our system is the data and associated profile that describes a computation to be performed. Since users may want to submit a large number of jobs by employing parameter sweeps or N-body calculations, HTCaaS introduces a concept of the Meta-Job which specifies a higher-level job description based on the OGF JSDL standard. Once a Meta-Job is submitted, HTCaaS automatically splits it into many jobs and inserts them into the Job Queue (implemented in ActiveMQ) managed by the Job Manager. All of required input data and produced results are stored at the User Data Manager. Once jobs are submitted into our system, agents (implemented in Java) are dispatched from Agent Manager and process jobs in Supercomputers, Grids, and Clouds. HTCaaS employs agent-based multi-level scheduling & streamlined job dispatching so that a first-level request to a batch scheduler (e.g., Load Leveler in PLSI Supercomputers, gLite for Grids, PBS for Amazon EC2) reserves resources by submitting agents as batch jobs and then each agent proactively pulls the tasks from the Job Manager which implements the lightweight and fast job dispatching mechanisms.

Therefore, users of HTCaaS are able to submit and execute hundreds of thousands of jobs (which can be simply expressed by a single JSDL script) within an automated process, effectively monitor them and process the final results. For those who are not familiar with XML style of scripting, we also provide an easy-to-use GUI tool which can automatically generate JSDL script based on user’s input so that it can be submitted into our system. The overall steps of job submission and execution in HTCaaS system are as followings:


JobSubExecSteps


  1. User logins HTCaaS and uploads input data through User Data Manager.
  2. User submits a Meta-Job (written in JSDL) which can be composed of multiple tasks.
  3. HTCaaS automatically divides a Meta-Job into multiple tasks based on the specification and insert them into the Job Queue.
  4. Agent Manager dispatches agents based on job requirements and resource availability.


Comparison: HTCaaS vs. Condor vs. DIRAC

Publications

Articles in Refereed Journals

  1. Cao Ngoc Nguyen, Jaehwan Lee, Soonwook Hwang, Jik-Soo Kim, On the role of message broker middleware for many-task computing on a big-data platform, Springer Cluster Computing: The Journal of Networks, Software Tools and Applications, Volume 22, Supplement 1, pp 2527–2540, January 2019.
  2. Jik-Soo Kim, Bui The Quang, Seungwoo Rho, Seoyoung Kim, Sangwan Kim, Vincent Breton, Soonwook Hwang, Towards Effective Scheduling Policies for Many-Task Applications: Practice and Experience based on HTCaaS, Concurrency and Computation: Practice and Experience, Volume 29, Issue 21, November 2017.
  3. Cao Ngoc Nguyen, Soonwook Hwang, Jik-Soo Kim, Making a Case for the On-demand Multiple Distributed Message Queue System in a Hadoop Cluster, Springer Cluster Computing: The Journal of Networks, Software Tools and Applications, Volume 20, Issue 3, September 2017.
  4. Cao Nguyen, Jik-Soo Kim, Jaehwan Lee, Soonwook Hwang, A Case Study of leveraging High-Throughput Distributed Message Queue System for Many-Task Computing on Hadoop, 5th International Workshop on Autonomic Management of high performance Grid and Cloud Computing (AMGCC’17), September 2017
  5. Jik-Soo Kim, Cao Nguyen, Soonwook Hwang, MOHA: Many-Task Computing meets the Big Data Platform, IEEE 12th International Conference on eScience (eScience 2016), October 2016.
  6. Cao Nguyen, Jik-Soo Kim, Soonwook Hwang, KOHA: Building a Kafka-based Distributed Queue System on the fly in a Hadoop cluster, 2016 IEEE International Conference on Cloud and Autonomic Computing (ICCAC) (from AMGCC’16 Workshop), September 2016.
  7. Md. Azam Hossain, Cao Ngoc Ngyuen, Jik-Soo KIm, Soonwook Hwang, Exploiting Resource Profiling mechanism for Large-scale Scientific Computing on Grid, J of Cluster Computing, 2016, Accepted for publication
  8. Eunji Hwang, Seontae Kim, Jik-Soo Kim, Sooonwook Hwang, Yourng-ri Choi, On the Role of Application and Resource Characterizations in Heterogeneous Distributed Computing System, J. of Cluster Computing, 2016, Accepted for publication
  9. Eunji Hwang, Suntae Kim, Tae-kyung Yoo, Jik-Soo Kim, Soonwook Hwang, and Young-ri Choi, Resource Allocation Policies for Loosely Coupled Applications in Heterogeneous Computing Systems, IEEE Transactions on Parallel and Distributed Systems, Vol. 27, No. 8, PP. 2349-2362, July 2015
  10. Jieun Choi, Younsun Ahn, Seoyoung Kim, Yoonhee Kim and Jaeyoung Choi, VM auto-scaling methods for high throughput computing on hybrid infrastructure, Springer Cluster Computing, Volume 18, Issue 3, September 2015
  11. Jik-Soo Kim, Beomseok Nam, and Alan Sussman, Scalable and effective peer-to-peer desktop grid system, Springer Cluster Computing, July 2014 (DOI10.1007/s10586-014-0390-z)
  12. Jik-Soo Kim, Seok-Kyoo Kim, Sangwan Kim, Seungwoo Rho, Seoyoung Kim, and Soonwook Hwang, An Analysis of Multi-level Scheduling Mechanism for Large-scale Scientific Computing, Journal of KIISE: Computing Practices and Letters, Volume 20, Number 7, July 2014.
  13. Seok-kyoo Kim, Jik-Soo Kim, Sangwan Kim, Seungwoo Rho, Seoyoung Kim, and Soonwook Hwang, HTCaaS(High Throughput Computing as a Service) in Supercomputing Environment, Journal of the Korea Contents Association, Volume 14, Number 5, May 2014
  14. Seoyoung Kim, Jik-Soo Kim, Soonwook Hwang, and Yoonhee Kim, Towards effective science cloud provisioning for a large-scale high-throughput computing, Springer Cluster Computing, December 2014
  15. Jik-Soo Kim, Seok-Kyoo Kim, Sangwan Kim, Seungwoo Rho, Seoyoung Kim, and Soonwook Hwang, High-Throughput Computing over Distributed Supercomputing Infrastructures: Technologies and Challenges, Journal of Next Generation Information Technology (JNIT), Volume 4, Number 8, October 2013.
  16. Jik-Soo Kim, Sangwan Kim, Seokkyoo Kim, Seoyoung Kim, Seungwoo Rho, Ok-Hwan Byeon, and Soonwook Hwang, Towards a Next Generation Distributed Middleware System for Many-Task Computing, International Journal of Software Engineering and Its Applications, Volume 7, Number 4, pages 379-389, July 2013.
  17. TTH Nguyen, HJ Ryu, SH Lee, S Hwang, V Breton, JH Rhee, D Kim, Virtual screening identification of novel severe acute respiratory syndrome 3C-like protease inhibitors and in vitro confirmation, Bioorganic & medicinal chemistry letters 21 (10), 3088-3091, May 2011.
  18. TTH Nguyen, HJ Ryu, SH Lee, S Hwang, J Cha, V Breton, D Kim, Discovery of novel inhibitors for human intestinal maltase: virtual screening in a WISDOM environment and in vitro evaluation,Biotechnology letters 33 (11), 2185, November 2011.

Articles in Refereed Conferences and Workshops

  1. Cao Nguyen, Jik-Soo Kim, Jaehwan Lee, Soonwook Hwang, A Case Study of leveraging High-Throughput Distributed Message Queue System for Many-Task Computing on Hadoop, 5th International Workshop on Autonomic Management of high performance Grid and Cloud Computing (AMGCC’17), September 2017
  2. Jik-Soo Kim, Cao Nguyen, Soonwook Hwang, MOHA: Many-Task Computing meets the Big Data Platform, IEEE 12th International Conference on eScience (eScience 2016), October 2016.
  3. Cao Nguyen, Jik-Soo Kim, Soonwook Hwang, KOHA: Building a Kafka-based Distributed Queue System on the fly in a Hadoop cluster, 2016 IEEE International Conference on Cloud and Autonomic Computing (ICCAC) (from AMGCC’16 Workshop), September 2016.
  4. Md Azam Hossain, Hieu Trogn Vu, Jik-soo Kim, and Soonwook Hwang, SCOUT: A Monitor & Profiler of Grid Resources for Large-Scale Scientific Computing, 3rd International Workshop on Autonomic Management of Grid and Cloud Computing (AMGCC’15) held with IEEE CAC 2015, September 2015.
  5. Jieun Choi, Seoyoung Kim, Theodora Adufu, Soonwook Hwang and Yoonhee Kim, A Job Dispatch Optimization Method on Cluster and Cloud for Large-scale High-Throughput Computing Service,3rd International Workshop on Autonomic Management of Grid and Cloud Computing (AMGCC’15) held with IEEE CAC 2015, September 2015.
  6. Eunji Hwang, Seontae Kim, Tae-Kyung Yoo, Jik-Soo Kim, Soonwook Hwang and Young-Ri Choi, Performance Analysis of Loosely Coupled Applications in Heterogeneous Distributed Computing Systems,3rd International Workshop on Autonomic Management of Grid and Cloud Computing (AMGCC’15) held with IEEE CAC 2015, September 2015.
  7. Suntae Kim, Eunji Hwang, Tae-Kyung Yoo, Jik-Soo Kim, Soonwook Hwang and Young-Ri Choi, Platform and Co-runner Affinities for Many-Task Applications in Distributed Computing Platforms, 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2015), May, 2015
  8. Bui The Quang, Jik-Soo Kim, Seungwoo Rho, Seoyoung Kim, Sangwan Kim, Soonwook Hwang, Emmanuel Medernach and Vincent Breton, A Comparative Analysis of Scheduling Mechanisms for Virtual Screening Workflow in a Shared Resource Environment, 2015 Workshop on Clusters, Clouds and Grids for Life Sciences (CCGrid-Life 2015) held with IEEE/ACM CCGrid 2015
  9. Jik-Soo Kim, Seungwoo Rho, Minho Lee, Seoyoung Kim, Sangwan Kim, and Soonwook Hwang, Large-Scale Drug Repositioning Simulation based on HTCaaS, Korea Computer Congress 2014], June 2014 [Best Paper Award]
  10. Sangwan Kim, Seungwoo Rho, Seoyoung Kim, Jik-Soo Kim, and Soonwook Hwang, An Implementation of HTCaaS User Web Portal: Easy Start of HTCaaS, A International Conference on Convergence Content (ICCC) 2014, June 2014.
  11. Seungwoo Rho, Jik-Soo Kim, Sangwan Kim, Seoyoung Kim, and Soonwook Hwang, A Scalability Performance Study for General-Purpose Applications on HTCaaS :The Database Perspective, A International Conference on Convergence Content (ICCC) 2014, June 2014.
  12. Jik-Soo Kim, Seungwoo Rho, Seoyoung Kim, Sangwan Kim, Seok-KyooKim, and SoonwookHwang, Large-Scale Scientific Simulations throughout HTCaaS: Technologies, Practice and Applications, International Symposium on Grids and Clouds 2014 (ISGC 2014), March 2014. (PDF)
  13. Jik-Soo Kim, Seungwoo Rho, Seoyoung Kim, Sangwan Kim, Seokkyoo Kim, and Soonwook Hwang, HTCaaS: Leveraging Distributed Supercomputing Infrastructures for Large-Scale Scientific Computing, ACM 6th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS'13) held with SC13, November 2013.
  14. Soonwook Hwang, Seoyoung Kim, Jik-Soo Kim, Seungwoo Rho, Sangwan Kim, Seokkyoo Kim, HTCaaS: Efficient and Simplified Large-Scale Scientific Computing over Supercomputers, Grids and Cloud, Research Poster at ACM Cloud and Autonomic Computing Conference (CAC 2013), August 2013. (PDF)
  15. Seoyoung Kim, Jik-Soo Kim, Soonwook Hwang and Yoonhee Kim, An Allocation and Provisioning Model of Science Cloud for High Throughput Computing Applications, 1st International Workshop on Autonomic Management of Grid and Cloud Computing (AMGCC’13) held with ACM CAC 2013, August 2013. (PDF)
  16. Jik-Soo Kim, Beomseok Nam and Alan Sussman, Autonomic Load Balancing Mechanisms in the P2P Desktop Grid, 1st International Workshop on Autonomic Management of Grid and Cloud Computing (AMGCC’13) held with ACM CAC 2013, August 2013. (PDF)
  17. Jik-Soo Kim, Seokkyoo Kim , Sangwan Kim, Seungwoo Rho, Seoyoung Kim, and Soonwook Hwang, Leveraging Distributed Supercomputing Infrastructures to support Large-Scale Scientific Computing, 3rd International Conference on Convergence Technology (ICCT’2013), July 2013.
  18. Jik-Soo Kim, Sangwan Kim, Seokkyoo Kim, Seoyoung Kim, Seungwoo Rho, Ok-Hwan Byeon, and Soonwook Hwang, From High-Throughput Computing to Many-Task Computing: Challenges, Systems and Applications, 2nd International Conference on Software Technology (SoftTech 2013), April 2013.
  19. Sangwan Kim, Seoyoung Kim, Seungwoo Rho, Seokkyoo Kim, Jik-Soo Kim and Soonwook Hwang, HTCaaS, a Viable Choice for Efficient and Simplified Large-Scale Scientific Computing, Research Poster at YongPyong International Winter Conference on Particle Physics (YongPyong-2013), February 2013.
  20. Seungwoo Rho, Seoyoung Kim, Sangwan Kim, Seokkyoo Kim, Jik-Soo Kim and Soonwook Hwang, HTCaaS: A Large-Scale High-Throughput Computing by Leveraging Grids, Supercomputers and Cloud, Research Poster at International Conference for High Performance Computing, Networking, Storage and Analysis (SC12), November 2012. (PDF)
  21. Sehoon Lee, Seokkyoo Kim, Seungwoo Rho and Soonwook Hwang, HTCaaS (HTC as a Service): A Large-scale HTC Problem Solving Environment Using Distributed and Heterogeneous Infrastructures, 2012 International Symposium on Grids and Clouds (ISGC), Feb 2012. (PDF)

Patent Applications

  1. Seoyoung Kim, Eunkyu Byun, Soonwook Hwang, Seokkyoo Kim, Jik-Soo Kim, Sangwan Kim, and Seungwoo Rho, METHOD AND APPARSTUS FOR ALLOCATING RESOURCE REFLECTING ADAPTIVE EVALUATION IN CLOUD COMPUTING FOR HIGH-THROUGHPUT COMPUTING (14/326,618), July 2014, U.S.A.
  2. Jik-Soo Kim, Seungwoo Rho, Seokkyoo Kim, Sangwan Kim, Soonwook Hwang, APPARATUS AND METHOD FOR ALLOCATING COMPUTING RESOURCE TO MULTIPLE USERS (10-2013-0087452), July 2013, Republic of Korea
  3. Seoyoung Kim, Soonwook Hwang, Seokkyoo Kim, Jik-Soo Kim and Eunkyu Byun, METHOD AND APPARATUS FOR JOB PROFILING FOR HIGH THROUGHPUT COMPUTING (10-2013-0080181), July 2013, Republic of Korea
  4. Seoyoung Kim, Soonwook Hwang, Sangwan Kim, Seungwoo Rho and Eunkyu Byun, METHOD AND APPARATUS FOR ALLOCATING RESOURCE REFLECTING ADAPTIVE EVALUATION IN CLOUD COMPUTING (10-2013-0080182), July 2013, Republic of Korea

Downloads & Demos

HTCaaS CLI

HTCaaS GUI

HTCaaS Web Portal

Start HTCaaS

  • Web portal - Only for resgistered user
    • To register
    • If you want to register HTCaaS, please click the above link and e-mail to htcaas-admin[at]kisti.re.kr
  • QnA(launchpad)
  • PLSI Registration (PDF) (Kr)

Utility

In details

If you have questions about HTCaaS, e-mail to htcaas-admin[at]kisti.re.kr