<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1875728222659097&amp;ev=PageView&amp;noscript=1">

30M Cell LES on a Desktop Computer

Home » Blog » 30M Cell LES on a Desktop Computer

Eastern Canada's Fundy Tidal Region (FTR) is one of the world's best and most energetic areas for tidal energy development. However, this highly variable ocean environment presents a number of challenges to both manufacturers and developers. Machines of any type face a particularly taxing environment, and as such, designers must account for site-specific conditions in their devices, modelling unsteady three-dimensional turbulent flow in complex tidal zone terrains.

This study analyzes two working examples from the FTR, where a client needed critical environmental flow information within 1 month on a strict budget. Conventional CFD tools and experimental testing would have far exceeded both deadline and budget. This article highlights how new highly-parallel heterogeneous manycore CFD solutions provided an accurate method of arriving at a critical engineering solution quickly and within budget. The growth rate and overall success of the global tidal marine renewable energy industry depends on innovation such as this, and cost-effective high-performance CFD provides an affordable and efficient way of augmenting an experimental program, as shown in this study.

The nature of the ocean environment makes the experimental field data (currently the primary source of information used by designers) both costly and difficult to obtain. A key example of this is the Acoustic Doppler Current Profiler (ADCP), widely used to measure flow velocity at a planned installation site. While it provides vertical mean velocity profile information critical to the design and operation of turbines, it only provides limited resolution for one geographic location. To acquire additional information demands numerous ADCP measurements, which cost between $30,000-$60,000, each.

The Sites

There are two sites used in this study to highlight different CFD approaches to accommodate local features, both approaches employing a GPU accelerated solution. The first is a large-scale tidal energy development site (Minas Passage) and the second a smaller community based development site (Grand Passage), both locations shown in Fig. 1. Both sites are part of the Bay of Fundy Tidal Region (FTR) where numerous tidal energy resources are being considered for development.

Screen Shot 2017-10-04 at 20.14.00.pngFig. 1 Location of important tidal energy sources in the Nova Scotia Bay of Fundy region, with the Minas Channel (Passage) and Grand Passage locations highlighted.

The Minas Passage - A classic example of the challenging ocean environment. This channel connects the Bay of Fundy to the Minas Basin. The world's highest tides are experienced here, and surface flow speeds reach 5 m/s. The Fundy Ocean Research Center for Energy (FORCE) has set up a 4 square km, 20–50 m deep region in the Minas Passage that provides tidal turbine developers with grid-connected, pre-approved sites upon which to test their devices. The FORCE site poses specific challenges due to the complex bathymetry surrounding it (visible in Fig. 2 top view). A large volcanic platform rises 10–15 m above the surrounding seafloor and generates high levels of turbulence on its leeward side. In addition, a nearby island influences the background turbulence level toward the northern third of the site on the ebb tide.

Screen Shot 2017-10-04 at 20.32.31.png Screen Shot 2017-10-04 at 20.32.37.png
Fig 2.
 In the left-hand view color shaded bathymetry of the FORCE region (within the Minas Passage) used to generate the right-hand boundary of the CFD domain, the red box denotes the FORCE Crown Lease Area, the yellow box the location of the computational domain. The orange dot indicates the location of the field placed ADCP device used for validation. Ebb tide flow, investigated in this study, comes from the south-east; due north is up. Shading corresponds to depth; areas shaded in green are above mean sea-level. In the bottom view the Grand Passage location with yellow box showing the computational domain and the orange dot the ADCP device location. 

The Grand Passage - Located between Brier Island and Long Island at the mouth of the Bay of Fundy, the passage is approximately 4km long and varies in width from 800 m to 2km. Water depth along the channel centre ranges from 10 to 30 m and the depth averaged speeds can reach 3 m/s within a tidal range of about 5 m. There is also an island at one end of the channel that generates significant eddies. Unlike the Minas Passage example, the resource is confined to a narrow passage with complex shorelines to consider along with the bathymetry. The CFD model must therefore be able to accommodate both the bathymetry and its intersection with the shoreline.

The unsteady turbulent flow field in these tidal sites is an essential piece of information for successful deployments. To fully characterize this turbulence requires a time resolved 3D Navier-Stokes solution at resolutions relevant to both devices and bathymetry. Tidal CFD studies are now employing Large-Eddy-Simulations (LES) or Detached-Eddy-Simulation (DES) for turbulence modelling to obtain this unsteady information.

The focus of this information is quite diverse, including;

  • synthetic turbulence modelling for improved far-field boundary conditions
  • structural loading due to turbulence 
  • interactions between vortex structures in a turbine array
  • unsteady wake/bathymetry interactions
  • enhanced turbine models

In the majority of cases, a compromise is made between the device level resolution of the unsteady flow field and the extent of the tidal flow environment that is included. An example of this is in the study of interacting turbine wakes, where the domain is limited to a 'simulation box' with synthetic turbulence to introduce turbulence present in the larger domain. This approach actually filters out large scale intermittent structures that can prove influential in the long term survivability of a turbine.

This Study

One of the long term goals for tidal CFD simulations is for approaches that can model flow fields at device scales (∼1 m) while including the wider flow context, which can be of the scale of kilometers. The range of scales leads to a very demanding environmental simulation problem in both computation time and hardware costs. The present study employing spatial parallelization on GPUs is one step toward addressing the need for device scale environmental simulations.

In the results presented by this study, domains are in the order of 1–2 km with mesh resolution at ∼1 m. Further speed-up of similar tidal flow simulations is being investigated using manycore space-time parallelization.

Specifically this study contributes to the literature on device-scale high-resolution CFD modelling in tidal zones through two very different tidal energy developments. Details of the simulation parameters with a particular focus on the meshing with bathymetry and shoreline, inlet boundary condition and synthetic turbulence, and efficient solution times are also discussed. 

The post-processing methodology developed for validation of simulation results to ADCP field data is also summarized. In addition, the study demonstrates good agreement between field data and simulation results.


The present study applied a refined CFD mesh, of ∼1–2 m resolution, with resolved bathymetry from the FORCE region in the Minas Passage to simulate a turbulent ebb tidal flow (at two tide states) using a Detached Eddy Simulation technique. The same level of mesh resolution and turbulence modeling was applied to the Grand Passage tidal flow but for ebb and flood and a more complex shoreline including an island. In both instances the simulations were conducted using the EXN/Aero manycore CFD solver, which was able to produce statistically converged LES results for meshes in the range of 25–30 million control volumes within one month on a desktop system comprised of two K40 NVIDIA GPUs.

2017-10-04 16_24_06-pagination_OE_4566-1[1].pdf.pngFig 3 (Left). Predicted instantaneous velocity field for ebb conditions, slice 5 m below sea level. Location of 4-beam ADCP indicated by the black dot near the top of the image. Fig 4 (Right). Predicted instantaneous velocity field for flood conditions, slice 5 m below sea level. Location of 4-beam ADCP indicated by the black dot near the top of the image.

For both the Minas Passage and Grand Passage locations the solutions were aided by using engineered inlet boundary conditions. This included interpolated velocity and wall shear stresses from FTR FVCOM simulations and synthetic turbulence to minimize domain-size requirements for boundary layer development. For both locations a virtual ADCP (VADCP) method of arranging and post-processing simulation monitor points was employed to mimic the processing methodology of ADCP devices. The change in post-processing methodology was found to have a significant influence on validating the CFD results against ADCP measurements. Simulation results were validated against field collected ADCP data at both tidal locations and the agreement was found to be good considering the complexity of the flow environment.

The paper outlines a way to improve data collection in the CFD simulation to better align predictions with ADCP measurements to help in validating the model. The deployment of the CFD simulation using a code that can take advantage of significantly more parallelism (by combined CPU and GPU usage) reduces the simulation times to a practical duration to support design and deployment. 

Important to this is a requirement that sufficient boundary condition information is available particularly in terms of quality bathymetry and larger regional model simulation results. Also important is the availability of physical ADCP measurements to undertake validation steps prior to extending the use of CFD results. This is important since each tidal energy site will have its own unique characterization issues and so a CFD model validated at one site should not be assumed to be validated for another. The next steps in the CFD model development will involve embedding turbines (with simplified models for rotating components) for virtual in-situ prototyping of turbines and other hydrokinetic devices.

What This Means For The Industry

Governments and organizations around the world are expressing keen interest in tidal power as a reliable energy resource, and it's not difficult to see why. However, the growth rate of this industry depends on sustained commitment to research and cost reduction in a number of areas. These will be best assessed on a site-by-site basis and include;

  • device installation and foundation costs
  • prime-mover component costs
  • operations and maintenance budgets

As mentioned earlier in this article, an ADCP only provides a limited resolution for one location, meaning numerous ADCP measurements are required, thus increasing costs.

Augmenting with high-resolution CFD simulations has been clearly highlighted as a way to improve the return on investment for an experimental program. A small number of strategically placed experimental measurements can be used to validate a CFD solution for a particular region providing detailed, time varying three-dimensional data. The extent of a validated region is difficult to quantify; however, a validated region is inherently greater than point ADCP measurements and naturally increases with the number of measurement devices included in the validation. 

In addition, validated simulations can be used as a predictive tool for testing arrangements of devices, as would be seen in a tidal turbine farm, without the need for additional experiments. 

The use of manycore parallelization for the CFD solution in this study shows the potential for deploying high-resolution CFD with turbines in-situ along with the wider tidal flow field. The efficiency of the simulations offers the potential of ∼100M node simulations obtained within time frames for supporting design iterations and using desktop oriented systems.

High-resolution CFD simulations have been available for several years now, but their role as a design tool has been limited by the high cost of supercomputing equipment, which have been primarily deployed for multicore (CPU based) coarse-grained parallelization, and the time needed to arrive at an engineering solution.

Recent studies, including this one, are demonstrating that new highly-parallel heterogeneous manycore (GPUs working in concert with CPUs) CFD solutions are making results available within industrially-relevant time frames, using far less costly supercomputing equipment. We explored how such a solution, notably one that is cloud-hosted, helps to overcome traditional limitations in a recent article. This trend toward increasingly powerful low-cost supercomputers at desktop scale is driven by exascale computing demands where new energy efficient architectures are a requirement. 

Such exascale trends are likely to have a huge impact on the capability of CFD in the future, and current trends in HPC have already begun. We recently covered the proactive approach needed to embrace such trends in a recent article. The use of GPU-oriented architectures for scientific computing is the first entry in a long-term trend, that will see complex new computing systems geared towards maximizing floating point operations at the lowest possible energy input. CFD software must be ready to embrace these trends.

In summary, this study highlights how cost-effective, high-performance CFD software could shape the future of tidal energy development, by providing an accurate, affordable, and efficient way of augmenting an experimental program.


Click here to view the study in full.

Try EXN/Aero For Yourself

EXN/Aero is a general purpose CFD cloud solver committed to embracing the needs of the engineering community. 

The EXN/Aero platform includes a meshing tool, the solver, and a post-processing tool, and plenty of storage for your files. There are a range of on-demand options available to users, helping them to better manage project cash flow. This affordable CFD software is sure to be an asset to companies or CFD freelancers like. We offer free trials of the product, starting with a walkthrough with one of our engineers, and including 10 hours of compute time.

Request Demo

2017-10-11 | Categories: CFD, simulations, HPC