(Click here for printable version)
Click on the quick recommendation section title
("Processors, "Networking", etc.) for detailed discussion on that topic:
||Fewer cores with higher clock speeds are preferable to a larger number of cores with lower clock speeds. Using higher clock speed cores also saves money as GridLink is licensed per core. Current generation Intel E5-26xx V3 processors with up to 14 cores per processor are recommended (16 and 18 core E5 processor options are not recommended). Servers containing 4 or more processors (Xeon E5-46xx or similar) are also not recommended.
||Minimum 1 Gigabit networking connecting all GridLink servers. All servers should be located in a single physical location on a single subnet without routers or firewalls between them.
||Virtualization of GridLink is not recommended. In cases where there is no other choice (such as public cloud environments), only specific node types and configurations should be used. See detailed discussion for further details.
||4 GB of memory per processor core - a 16 core server would need 64 GB ram, 24 cores would need 96 GB, etc.
||Windows Server 2012 R2 Standard is recommended, 2012 R1 and 2008 R2 Enterprise are also supported. 2008 R2 Standard is not recommended due to limitations on maximum amount of memory per server.
||Local disk performance on all servers should be equivalent to 3 x 15k drives in raid 0 for servers up to 24 cores. 4 x 15k drives in raid 0 are required in servers with more than 24 cores. The use of raid 10 for redundancy is recommended on master servers but is not required on helper servers due to automatic recovery features in GridLink. Using NAS or SAN devices for GridLink local storage is not recommended.
Antivirus, Backups, etc.:
||If antivirus is required, proper exclusions must be in place for all servers. In addition, other applications (backup, deduplication, etc.) which can interfere with disk access should not be run on GridLink servers.
Master and Helper Servers:
||Each queue in your GridLink farm requires an individual server - it is not possible to have more than one queue on a single server. As a result, the number of servers in your farm must be greater than or equal to the number of queues.
||No data is stored permanently on the GridLink servers. As a result, only the GridLink farm profile needs to be back up. GridLink offers a built in option to backup this profile to a location outside the Grid environment.
||GGY recommends deploying AXIS EnterpriseLink along with GridLink to provide frontend access to the GridLink environment
vendors (IBM, HP, Dell etc.) offer rack servers designed for the Xeon E5-2600
is a good example of a 2U server that supports two Xeon E5-2600
series chips and
up to eight
2.5" 15K SAS drives.
||Configuration 1 - 16 Cores
||Configuration 2 - 24 Cores
||Configuration 3 - 28 Cores
Xeon E5-2667 v3
Xeon E5-2690 v3 (2.6 Ghz)
||Xeon E5-2697 v3 (2.6
|Drives - Master:
||6x300 GB 15k SAS, RAID 10
||6x300 GB 15k SAS, RAID 10
||8x300 GB 15k SAS, RAID 10
|Drives - Helpers:
||3x300 GB 15k SAS, RAID 0
||3x300 GB 15k SAS, RAID 0
||4x300 GB 15k SAS, RAID 0
Sample Configuration Notes:
Configure additional servers to get to the required farm size - i.e. 6 x Configuration 1 would produce a farm of 96 cores.
A minimum of one server per queue is required - a farm with 4 queues, for example, would require a minimum of 4 servers. See "Master and Helper Servers" for more details.
It is generally recommended to use the same configuration for all nodes in a farm. If mixing different server configurations, use the fastest (highest clock speed) nodes as masters.
The hard drives specifications are minimums and performance may be improved with faster drives, SSD drives or more drives per RAID array.
Blade servers can be used, but tend to have limitations on local storage. See "Blade Servers" for more details.
The Intel E5-26xx V3
processors carry our top recommendation for use with GridLink. Processors
are available with 4, 6, 8, 10, 12, 14, 16 and 18 cores per processor in two
processor servers. In general, the higher the core clock speed the
better each core will perform. Sixteen 3.4 GHz cores, for example,
will outperform sixteen 2.6 GHz cores. 3.4 GHz clock speeds, however, are only
available on 6 core processors while 2.6 GHz processors are available with 14
cores. So you would need more than twice as many 3.4 Ghz processors to get
the same number of cores. As a result, other factors such as server and
operating system costs, licensing fees, and your datacenter density and power
consumption requirements should be considered along with your speed requirements
to determine the optimal processors for your environment.
Note that we do not recommend 16 and
18 core processor options. The maximum native clock speed of 2.3 GHz combined with
the high price and power consumption of these chips makes them a poor choice for
Our top recommendations
are as follows:
|Xeon E5-2697 V3
||2.60 GHz 14
||2.60 GHz 12
|Xeon E5-2685 V3
||2.60 GHz 12
|Xeon E5-2660 V3
||2.60 GHz 10
|Xeon E5-2667 V3
||3.20 GHz 8 cores
We are disappointed by the performance of 4-processor and 8-processor servers
for high performance computing so at this time we will not recommend their use
in a GridLink farm.
Connect up all
the servers in your GridLink farm on the same subnet using at least 1 Gbps networking. This
applies even if the rest of your data centre is using a 100Mbps network.
InfiniBand or 10G Ethernet is preferred but not required and will have the most
benefit on the Master servers.
At this time we require a
registry entry to disable SMB 2.0 which automatically downgrades to SMB1 level.
AXIS GridLink Advisory - Windows 2008 with SMB
2.0 Severely Impacting AXIS GridLink for more information.
We do not recommend virtualizing GridLink servers as the
hypervisor layer reduces the performance of the hardware. We do, however, support virtual
machines in those environments where there is no direct access to “bare metal”
hardware, such as public clouds.
This support is
conditional on meeting the following minimum requirements:
Virtual servers are appropriate for EnterpriseLink Servers or Front-End Servers if adequate
processing power and memory as per GGY requirements for each concurrent user
session is backed up by the identical number of CPU cores and the amount of RAM
in the host machine.
- There is one-to-one
correspondence between physical (non-hyperthreaded) cores on the host and
virtual cores allocated to the VM running GridLink.
- There is a minimum 4GB
per core of physical RAM on the host backing the RAM allocated to the VM.
- The disk space allocated to the virtual machine
meets the requirements outlined in the "Hard Drives" section of this
- The performance of
the disks in the host is on a par with our recommended configuration for a
physical machine (min 3 way striping x 15 K, RAID 10)
- Minimum 1Gbps network
connectivity (single subnet)
- Performance and
stability can only be guaranteed if a single VM is hosted on a physical
server with one-to-one correspondence between the size of the host and the
size of the VM. If multiple VM’s are running on the same host the effect of
their load on the GridLink VM is unpredictable. Note: this requirement from
GGY is consistent with the approach chosen by all major cloud providers for
compute intensive (HPC) applications like AXIS.
Memory and O/S
You will need at least 4GB of memory per processor core.
Please note that each Windows version has its own memory limit. For details, please review Microsoft KB article "Memory Limits for Windows Releases".
For new installations we recommend Windows Server 2012 R2 Standard. Windows Server 2012 Standard and Windows Server 2008 R2 Enterprise are also supported. We no longer recommend Windows Server 2008 R2 Standard as it is limited to 32 GB of memory per machine. With modern 24+ core servers, this is not enough to meet our minimum requirement of 4 GB of memory per core. Window Server 2008 R2 Enterprise supports 2 TB of ram per machine and 2012 (R2) Standard supports up to 4 TB.
require access to
large amounts of high speed storage. This storage is temporary - it is
used during the calculations, but no data is stored there permanently. We
recommend the larger of either:
cores in the server> multiplied by <the uncompressed size of your largest dataset> + 25%
extra space as a buffer
Minimum disk performance should be the equivalent to 3 x 15k drives in Raid 0, with 4 x 15k drives in Raid 0 being recommended when using servers with more than 24 cores. If a disk failure occurs on a helper server, the work is automatically redistributed to other nodes, so disk redundancy on helper servers is optional. On master servers, however, it is recommended to provision 6 or 8 disks in Raid 10 so that disk failure does not cause running jobs to fail. You may also provide separate drives (RAID 1) for the O/S on both Master and Helpers servers although this is not a GridLink requirement or a GGY recommendation. You can save money and drive bays by introducing an O/S partition of say 100GB on the main RAID array.
We recommend that this storage be provided via local disks. For master servers only, you can alternately provision the drives via a direct fiber connection to a suitable SAN. The SAN, however, needs to be able to sustain performance equivalent to the local disk configurations noted above. Helper servers should not be provisioned with disks residing on a SAN. Given the temporary nature of the data stored on the helper servers, and the automatic recovery mechanisms, there is no advantage associated with the added expense of SAN based disks. In addition, we have seen situations where the combined throughput of a large number of helper servers has caused congestion and performance problems even on enterprise grade SAN devices.
We get a lot of questions about SSD devices instead of traditional spinning disks. They have improved a lot in terms of reliability and affordability. They are very fast for random access reads and writes, but this does not necessarily translate into improved AXIS performance since for most of the time we rely on sequential reads and writes where they offer little advantage. They are a good solution in situations like blade servers where space is limited and, as they continue to evolve, may eventually replace spinning disks as our recommended solution. At this time, however, we still do our development and testing against the performance characteristics of the spinning disk configurations listed above.
Important - Hard Disk Performance
Both master and helper servers perform a large number of time
sensitive disk reads and writes. As a result, it is vital that you not use any
software which interferes with native disk access. This software includes (but
is not limited to):
backup software (e.g. Microsoft Volume Shadow Copy Service)
anti-virus scanners (please see the "Anti-Virus" section of this document
for more info on anti-virus exclusions)
These types of
software will cause severe instability and performance issues.
Master Servers and Helper Servers
You can now
choose how many cores you want in a server farm and how many Queues you want on
that farm. Note that each Queue requires a separate server. The number of Queues
should correspond to how you wish to use the farm. For example, you may wish to
set up separate Queues for Pricing, Testing and Valuation. You can buy GridLink
Core licenses ($1000 per core) and GridLink Queue licenses ($4000 per Queue) to
support the configuration you require. Special pricing is available for Large
Farms (256 cores or greater).
A server with a
GridLink Queue license is a Master server, and without it, it's a Helper server. In most cases the requirements for master and helpers servers are the same except that we recommend redundant disk configurations on the masters but redundancy is optional on the helpers. If mixing different server configurations, the GridLink queue licenses should always be attached to the servers
with the fastest processors and best disk subsystems in the farm.
recommend that the datasets you run should be stored on a network share with
fast and reliable connection between the server hosting this share and the
Master servers in your farm. The GridLink service will automatically copy the
Dataset from its current location to the drive of the Master server before the
job starts, and copy it back once the job is complete. This is a change from our
earlier recommendation to store the datasets on the drive of the Master server.
The reason for this change is that the end user may be opening datasets located
on the Master Server’s drives and running jobs or performing other disk
intensive tasks locally on his/her desktop or front-end/EnterpriseLink server
before or after the GridLink run This may impose an additional load on the
GridLink server's drive system that can cause a significant deterioration in
performance or in some cases instability on both sides: the GridLink runs as
well as in the users’ interactive AXIS sessions.
Blades or Rack Mounted Servers
servers offer excellent performance and flexible storage options, but many Blade
servers provide very limited disk capacity, speed, and do not support the fast
processors. As a result, you need to be careful that the recommended storage, memory and
processors are available in the Blade chassis that you are considering. Some 1U
rack servers are also limited to two 3.5" drives or four 2.5" drives. We recommend 2U rack servers for their storage
abilities. If you are using Blade or other small servers you may get the disk
speed you need by using SSD drives or Fusion IO cards.
If you implement our current
recommendations for storing the datasets outside of the GridLink farm you no
longer have any data residing on the GridLink master’s or helper’s drives that
needs backing up. Taking a snapshot or image of the server to be able to recover
the operating system and installed executable files should be quite sufficient.
Even this may not be needed as re-installing GridLink is usually a very quick
process. You should consider backing up the farm profile after every
configuration change for faster recovery. See this page for more information:
partitions outside of the GridLink farm that house the datasets and other user
data files we now provide intelligent, safe and fully automated backup
functionality built into the AXIS EnterpriseLink module. For more information
please contact GGY System support.
Do not perform
backups using third party tools on GridLink Master or Helper servers while AXIS
is running under any circumstances. The backup you perform may lock files that
AXIS needs and produce errors and the backup will also be unusable since it may
be made while the dataset is changing.
that still house the datasets on the local drives of the Masters we have built
an intelligent backup facility into GridLink itself. It allows you to schedule
backups like other backup software but it will not attempt to back up a dataset
that is running - it will wait for a suitable time. The backup it produces is
simply a zip file for each Dataset or Database located in a safe place for you
to backup later using your normal backup software.
AXIS will store only temporary data on the Helper servers so there is no need to back these up.
You do not need
to backup the various AXIS versions installed on the GridLink servers because
they are always readily available for download from our website.
We recommend that you set up one or more AXIS EnterpriseLink front end servers to support the
interactive clients who are using your farm, and also for those who do not
require farm access. EnterpriseLink is included in your AXIS license.
For more information please visit
AXIS EnterpriseLink web site or