Home
In the one year since the MicroTCA specification, over 60 companies worldwide have developed AdvancedMC modules such as CPU boards, hard disk modules, graphics cards, DSP boards and GBit Ethernet switches. To ensure that all such MicroTCA components will work smoothly in one system, PICMG regularly organises, at the initiative of the companies, so-called Interoperability Workshops, in which developers from the participating manufacturers can test their MicroTCA components together in realistic conditions (Fig. 1).
Fig. 1. Example: Test rig
Why is this so important today? The need for such workshops began with the introduction of the AdvancedTCA standard itself. The design of these systems is highly complex, and becomes more so with the MicroTCA standard. Above all, communication and management within one system is substantially more comprehensive than in, for example, VMEbus or CompactPCI. The MicroTCA specification defines the management quite precisely, yet also leaves room for interpretation. At the workshops, individual component manufacturers can collaborate on the scope for interpretation available to them and establish standards. All participating companies are naturally interested in the ongoing development of the MicroTCA standard and seek to ensure that users will receive functioning systems.
MicroTCA management: an overview
AdvancedTCA defines a complete management function with the shelf manager as its central control element. This regulates the ventilation, power management and the individual AdvancedTCA blades. When AdvancedMC boards are used in AdvancedTCA systems, an additional management instance is necessary between the module management controller on the AdvancedMC module and the shelf manager: the carrier IPMC (Intelligent Platform Management Controller), situated on the AdvancedMC carrier. MicroTCA takes over the management structure from AdvancedTCA and defines the MicroTCA backplane, the MCHs and the power modules as MicroTCA carriers, analogous to the AdvancedMC carriers in the AdvancedTCA shelf. One MicroTCA shelf is a complete chassis and consists of up to 16 MicroTCA carriers, the ventilation and case. A MicroTCA system is then a network of multiple MicroTCA shelves. Thus there are also three instances of management controllers: the carrier manager, the shelf manager and the system manager.
MicroTCA carrier and shelf
In the AdvancedTCA system, two new modules are defined in MicroTCA for the functions that are taken over from the AdvancedMC carrier: the MicroTCA carrier hub (MCH) and the MicroTCA power module (PM), both as plug-in modules in the AdvancedMC format. In the cooling unit (CU) MicroTCA defines a further component at shelf level. This is responsible for cooling of the shelf.
MCH - The MicroTCA carrier hub is the "management centre" for all modules specified here and all AdvancedMC modules. The MCH has the same form factor as an AdvancedMC module. It also takes over the hub functionality in the carrier and is responsible for the clock generation and/or transmission.
PM – The power module is the power-supply module, likewise with the same form factor as an AdvancedMC module. It accepts a variety of AC and DC input voltages and delivers 12 V DC for the AdvancedMC modules and a separate 3.3 V DC for the management.
CU – The cooling unit ensures the ventilation of the shelf. Normally this takes the form of a hot swap-capable fan cassette unit. The fan cassette is defined at shelf level but is controlled by the shelf manager via the carrier manager.
Physical management connections
For the physical connection between the modules the I2C (Inter Integrated Circuit) bus is specified, on which the IPMI (Intelligent Platform Management Interface) platform management protocol runs. Today I2C is an industry standard for control, diagnostics and monitoring solutions in countless embedded applications. The connection between the MCH and the AdvancedMC modules is, as also with the AdvancedTCA carrier, implemented radially, that is, with a separate I2C bus (IPMB-L) between the MCH and each AdvancedMC module. Between the MCH, the CU and the PM, the I2C bus is configured as logical IPMB-0. The logical IPMB-0 is redundant and is physically split into an IPMB-A and an IPMB-B. There is additionally a special, local I2C bus on which the I2C protocol runs. This local I2C bus ties a SEEPROM element on the backplane to each MCH. Information specific to the carrier is stored in this element, which is known as the carrier FRU (field replaceable unit) data. It contains, for example, information on the bus connections that the backplane provides between the modules.
Management controllers
On the modules within the MicroTCA shelf sit the management controllers, connected to one another via the management buses, i.e. IPMB-L or IPMB-0. These management controllers differ according to the module type.
MCMC – MicroTCA Carrier Management Controller: This is located on the MCH and performs the control function of the AdvancedMC modules, the power modules and the cooling units via IPMB-L and IPMB-0. The MCMC receives information on the MicroTCA carrier via the I2C bus.
MMC – Module Management Controller: This is accommodated on the AdvancedMC module and is linked to the MCH via IPMB-L.
EMMC - Enhanced Module Management Controller for the cooling unit and the power module. Connection via IPMB-0 to the MCH.
Fig. 2 gives an overview of the relationships of the modules and management controllers. In this diagram modules such as MCH, CU and PM are not redundantly implemented. To obtain redundancy, the MicroTCA specification provides for up to two MCHs, up to two CUs and up to four PMs. The position of the shelf manager can also vary with the implementation; in this example it is outside the MicroTCA shelf, but it could also be implemented, for example, on an AdvancedMC module or an MCH.
MicroTCA management: an overview
AdvancedTCA defines a complete management function with the shelf manager as its central control element. This regulates the ventilation, power management and the individual AdvancedTCA blades. When AdvancedMC boards are used in AdvancedTCA systems, an additional management instance is necessary between the module management controller on the AdvancedMC module and the shelf manager: the carrier IPMC (Intelligent Platform Management Controller), situated on the AdvancedMC carrier. MicroTCA takes over the management structure from AdvancedTCA and defines the MicroTCA backplane, the MCHs and the power modules as MicroTCA carriers, analogous to the AdvancedMC carriers in the AdvancedTCA shelf. One MicroTCA shelf is a complete chassis and consists of up to 16 MicroTCA carriers, the ventilation and case. A MicroTCA system is then a network of multiple MicroTCA shelves. Thus there are also three instances of management controllers: the carrier manager, the shelf manager and the system manager.
MicroTCA carrier and shelf
In the AdvancedTCA system, two new modules are defined in MicroTCA for the functions that are taken over from the AdvancedMC carrier: the MicroTCA carrier hub (MCH) and the MicroTCA power module (PM), both as plug-in modules in the AdvancedMC format. In the cooling unit (CU) MicroTCA defines a further component at shelf level. This is responsible for cooling of the shelf.
MCH - The MicroTCA carrier hub is the "management centre" for all modules specified here and all AdvancedMC modules. The MCH has the same form factor as an AdvancedMC module. It also takes over the hub functionality in the carrier and is responsible for the clock generation and/or transmission.
PM – The power module is the power-supply module, likewise with the same form factor as an AdvancedMC module. It accepts a variety of AC and DC input voltages and delivers 12 V DC for the AdvancedMC modules and a separate 3.3 V DC for the management.
CU – The cooling unit ensures the ventilation of the shelf. Normally this takes the form of a hot swap-capable fan cassette unit. The fan cassette is defined at shelf level but is controlled by the shelf manager via the carrier manager.
Physical management connections
For the physical connection between the modules the I2C (Inter Integrated Circuit) bus is specified, on which the IPMI (Intelligent Platform Management Interface) platform management protocol runs. Today I2C is an industry standard for control, diagnostics and monitoring solutions in countless embedded applications. The connection between the MCH and the AdvancedMC modules is, as also with the AdvancedTCA carrier, implemented radially, that is, with a separate I2C bus (IPMB-L) between the MCH and each AdvancedMC module. Between the MCH, the CU and the PM, the I2C bus is configured as logical IPMB-0. The logical IPMB-0 is redundant and is physically split into an IPMB-A and an IPMB-B. There is additionally a special, local I2C bus on which the I2C protocol runs. This local I2C bus ties a SEEPROM element on the backplane to each MCH. Information specific to the carrier is stored in this element, which is known as the carrier FRU (field replaceable unit) data. It contains, for example, information on the bus connections that the backplane provides between the modules.
Management controllers
On the modules within the MicroTCA shelf sit the management controllers, connected to one another via the management buses, i.e. IPMB-L or IPMB-0. These management controllers differ according to the module type.
MCMC – MicroTCA Carrier Management Controller: This is located on the MCH and performs the control function of the AdvancedMC modules, the power modules and the cooling units via IPMB-L and IPMB-0. The MCMC receives information on the MicroTCA carrier via the I2C bus.
MMC – Module Management Controller: This is accommodated on the AdvancedMC module and is linked to the MCH via IPMB-L.
EMMC - Enhanced Module Management Controller for the cooling unit and the power module. Connection via IPMB-0 to the MCH.
Fig. 2 gives an overview of the relationships of the modules and management controllers. In this diagram modules such as MCH, CU and PM are not redundantly implemented. To obtain redundancy, the MicroTCA specification provides for up to two MCHs, up to two CUs and up to four PMs. The position of the shelf manager can also vary with the implementation; in this example it is outside the MicroTCA shelf, but it could also be implemented, for example, on an AdvancedMC module or an MCH.
Fig. 2. Overview of MicroTCA shelf
Shelf management hierarchy
The management controllers described here are physical units. They have a fixed position on their respective modules. There is, additionally, the actual - logical - management hierarchy.
On the lowest rung of this hierarchy is the carrier manager. This ensures that the AdvancedMC modules are activated and that only AdvancedMC modules with compatible software protocols are enabled. The carrier manager resides on the active MCH in its MCMC.
One hierarchical rung above this is the shelf manager. One or more carrier managers reports to this. It has no "fixed place" in the system. It is possible to implement the shelf manager on the MCH, but it can also be on an AdvancedMC module or outside the carrier. An important task for the shelf manager is control of the ventilation. Temperature sensors on all modules monitor the temperature constantly. If the alarm threshold on one of the modules is exceeded, the shelf manager receives via the carrier manager a signal, a so-called "event", and initiates an appropriate response, e.g. adjusting the fan speed accordingly.
The top rung of the hierarchy is occupied by the system manager. This administers multiple MicroTCA shelves. The MicroTCA specification does not, however, go into detail concerning the implementation of the system manager.
A workshop every year
The last MicroTCA Interoperability Workshop took place in late September 2007, for the first time in Europe, hosted by Schroff GmbH at its premises in Straubenhardt, Germany. More than 50 developers from 18 companies in Europe, the USA and Asia gathered for a four-day testing marathon and to exchange ideas. Each participant specifies at registration what equipment he will bring and what he wishes to test. From this information a test matrix is prepared in advance that describes in detail who wishes to test what and with whom. By way of example, two or three manufacturers will test IPMI transfer or data transfer using their own components (in the case of system, MCH and AdvancedMC manufacturers). The initiative continues beyond the test; where problems are detected, the participants work together on a solution until everything functions correctly.
Schroff tests its cooling unit
This cooling unit is an intelligent plug-in fan unit (Fig. 3) whose fan speed is controlled by the MicroTCA shelf manager. It has its own microcontroller, the so-called EMMC, which establishes the communication between the fans and sensors on the CU and the shelf manager.
The management controllers described here are physical units. They have a fixed position on their respective modules. There is, additionally, the actual - logical - management hierarchy.
On the lowest rung of this hierarchy is the carrier manager. This ensures that the AdvancedMC modules are activated and that only AdvancedMC modules with compatible software protocols are enabled. The carrier manager resides on the active MCH in its MCMC.
One hierarchical rung above this is the shelf manager. One or more carrier managers reports to this. It has no "fixed place" in the system. It is possible to implement the shelf manager on the MCH, but it can also be on an AdvancedMC module or outside the carrier. An important task for the shelf manager is control of the ventilation. Temperature sensors on all modules monitor the temperature constantly. If the alarm threshold on one of the modules is exceeded, the shelf manager receives via the carrier manager a signal, a so-called "event", and initiates an appropriate response, e.g. adjusting the fan speed accordingly.
The top rung of the hierarchy is occupied by the system manager. This administers multiple MicroTCA shelves. The MicroTCA specification does not, however, go into detail concerning the implementation of the system manager.
A workshop every year
The last MicroTCA Interoperability Workshop took place in late September 2007, for the first time in Europe, hosted by Schroff GmbH at its premises in Straubenhardt, Germany. More than 50 developers from 18 companies in Europe, the USA and Asia gathered for a four-day testing marathon and to exchange ideas. Each participant specifies at registration what equipment he will bring and what he wishes to test. From this information a test matrix is prepared in advance that describes in detail who wishes to test what and with whom. By way of example, two or three manufacturers will test IPMI transfer or data transfer using their own components (in the case of system, MCH and AdvancedMC manufacturers). The initiative continues beyond the test; where problems are detected, the participants work together on a solution until everything functions correctly.
Schroff tests its cooling unit
This cooling unit is an intelligent plug-in fan unit (Fig. 3) whose fan speed is controlled by the MicroTCA shelf manager. It has its own microcontroller, the so-called EMMC, which establishes the communication between the fans and sensors on the CU and the shelf manager.
Fig. 3. Standard fan cassette unit for a 48 HP MicroTCA system with cooling unit manager
The AdvancedMC modules built into the system (e.g. CPUs) are equipped with their own logic and with temperature sensors. If a temperature exceeds the preset threshold value, this information is passed on as an "event" to the shelf manager. The shelf manager then instructs the cooling unit to adjust the fan speed accordingly. In the opposite direction, the cooling unit reports the status of the fans and any failure or error function to the shelf manager. By this means a very high failure safety for the MicroTCA system is obtained.
The communications protocol employed is the IPMI - Intelligent Platform Management Interface required by the MicroTCA specification. This makes this plug-in cooling unit theoretically compatible with all shelf managers that conform to MicroTCA. This was verified at the Interoperability Workshop. The Schroff cooling unit was tested with MCHs of various manufacturers and proved a success. A further test was performed with various MCHs and complete Schroff systems including backplane. Here the purpose of the test was to check the information exchange between MCH and MicroTCA system at start-up. The test checked that the MCHs could read the chassis (FRU) data from the EPROMs integrated in the backplane, and whether all necessary data was present or whether the MCHs required further data. These details are not defined precisely in the specification and may thus be agreed, for example at a workshop of this kind, among the individual manufacturers. The testing of different MCH manufacturers also provided assurances that components may be combined, independently of manufacturer, into a given system.
Influencing the specification
The main purpose served by the workshops is to delimit the areas left open to interpretation by the specification among the various manufacturers. Quasi-standards are drafted that do not, however, necessarily become written into the specification. Only when individual tests indicate that certain parameters are a real cause of error function in MicroTCA components and systems is the problem raised in the next engineering change request of the specification.
The highest aim of the Interoperability Workshops is to ensure the functional readiness and implementation of the MicroTCA standard on a wide base. Users should be given the opportunity to obtain tested MicroTCA standard components from the widest range of manufacturers. Here, company policy and notions of competition play a subordinate role.
The communications protocol employed is the IPMI - Intelligent Platform Management Interface required by the MicroTCA specification. This makes this plug-in cooling unit theoretically compatible with all shelf managers that conform to MicroTCA. This was verified at the Interoperability Workshop. The Schroff cooling unit was tested with MCHs of various manufacturers and proved a success. A further test was performed with various MCHs and complete Schroff systems including backplane. Here the purpose of the test was to check the information exchange between MCH and MicroTCA system at start-up. The test checked that the MCHs could read the chassis (FRU) data from the EPROMs integrated in the backplane, and whether all necessary data was present or whether the MCHs required further data. These details are not defined precisely in the specification and may thus be agreed, for example at a workshop of this kind, among the individual manufacturers. The testing of different MCH manufacturers also provided assurances that components may be combined, independently of manufacturer, into a given system.
Influencing the specification
The main purpose served by the workshops is to delimit the areas left open to interpretation by the specification among the various manufacturers. Quasi-standards are drafted that do not, however, necessarily become written into the specification. Only when individual tests indicate that certain parameters are a real cause of error function in MicroTCA components and systems is the problem raised in the next engineering change request of the specification.
The highest aim of the Interoperability Workshops is to ensure the functional readiness and implementation of the MicroTCA standard on a wide base. Users should be given the opportunity to obtain tested MicroTCA standard components from the widest range of manufacturers. Here, company policy and notions of competition play a subordinate role.
