Atmospheric Modeling & Analysis Research
Distributed or so-called massively parallel processing, such as with the Community Multiscale Air Quality Model (CMAQ), enables the efficient computation of problems with very large problem sizes. However, programming for effective distributed processing requires careful code design and structuring. The basic principle of parallel computing is divide-and-conquer. This principle renders lower overall computational time (CPU) and potentially reduced local memory utilization (storage), and is achieved by decomposing the problem space into many smaller problems that are solved concurrently. In most applications, a processor must access data that reside in the memory of other processors.
The following is an example to illustrate this.
DO J = 1, N
DO I = 1, M
DATA(I,J) = A(I+2,J) * A(I, J-1)
Depending on the platform, there are two main data access paradigms: message passing and shared memory, and in some cases, a combination of both. Each paradigm has its pros and cons. It might be convenient, for example, for programmers to see a global memory model on a distributed processor system. However, interprocessor communication bandwidth might become a serious perfomance issue.
For the platforms we envision that will run the CMAQ models, message passing seems the best choice, and consequently we have developed some of the major codes, in particular the CMAQ Chemistry-Transport Model (CCTM), to use this paradigm.
The backbone of the inter-processor communication within CMAQ is the STENEX library which was developed inhouse and based on Message Passing Interface (MPI). STENEX library handles domain decomposition details as well as various types of inter-communication scheme, for example, (a) interior to ghost region, where the ghost region is indicated in light blue, (b) interior to interior, (c) subsection data redistribution, and (d) selective data collection.
( a ) interior to ghost region
( b ) interior to interior
( c ) redistribution of a subsection of data
( d ) selective data collection