This technical article demonstrates the concept of nested aggregation and how the dimensionality can be used within nested aggregation to achieve desirable results.
NOTE: The example depicted below can be partially followed within the Strategy Tutorial project if the fact is replaced by the fact. The resulting sql may be different but the sql generation logic is still valid.
Consider the dimension with the following structure:

The fact only exists at the lowest level of this hierarchy (). This means that there are no aggregate tables that contain an attribute ID higher than .
Given the following objects:
A report that has , and the average subtotaling enabled, will return:
Error in saving the cube, com.microstrategy.webapi.MSTRWebAPIException:() Hortonworks HDP 2.0 Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx Certified Cloudera CDH 5.0 and 5.1 Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx Certified MapR 3.x and 4.x Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx Certified Pivotal HD 2.0.1 and 2.1.0 Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx CertifiedBrowseDenyReadGrantWriteDefaultDeleteDefaultControlDefaultUseGrantExecuteGrantEXPLAIN OperatorsQuery Execution StepsDescriptionSCANSequential ScanscanAmazon Redshift relation scan or table scan operator or step. Scans whole table sequentially from beginning to end; also evaluates query constraints for every row (Filter) if specified with WHERE clause. Also used to run INSERT, UPDATE, and DELETE statements.JOINS: Amazon Redshift uses different join operators based on the physical design of the tables being joined, the location of the data required for the join, and specific attributes of the query itself. Subquery Scan -- Subquery scan and append are used to run UNION queries.Nested LoopnloopLeast optimal join; mainly used for cross-joins (Cartesian products; without a join condition) and some inequality joins.Hash JoinhjoinAlso used for inner joins and left and right outer joins and typically faster than a nested loop join. Hash Join reads the outer table, hashes the joining column, and finds matches in the inner hash table. Step can spill to disk. (Inner input of hjoin is hash step which can be disk-based.)Merge JoinmjoinAlso used for inner joins and outer joins (for join tables that are both distributed and sorted on the joining columns). Typically the fastest Amazon Redshift join algorithm, not including other cost considerations.AGGREGATION: Operators and steps used for queries that involve aggregate functions and GROUP BY operations.AggregateaggrOperator/step for scalar aggregate functions.HashAggregateaggrOperator/step for grouped aggregate functions. Can operate from disk by virtue of hash table spilling to disk.GroupAggregateaggrOperator sometimes chosen for grouped aggregate queries if the Amazon Redshift configuration setting for force_hash_grouping setting is off.SORT: Operators and steps used when queries have to sort or merge result sets.SortsortSort performs the sorting specified by the ORDER BY clause as well as other operations such as UNIONs and joins. Can operate from disk.MergemergeProduces final sorted results of a query based on intermediate sorted results derived from operations performed in parallel.EXCEPT, INTERSECT, and UNION operations:SetOp Except [Distinct]hjoinUsed for EXCEPT queries. Can operate from disk based on virtue of fact that input hash can be disk-based.Hash Intersect [Distinct]hjoinUsed for INTERSECT queries. Can operate from disk based on virtue of fact that input hash can be disk-based.Append [All |Distinct]saveAppend used with Subquery Scan to implement UNION and UNION ALL queries. Can operate from disk based on virtue of "save".Miscellaneous/Other:HashhashUsed for inner joins and left and right outer joins (provides input to a hash join). The Hash operator creates the hash table for the inner table of a join. (The inner table is the table that is checked for matches and, in a join of two tables, is usually the smaller of the two.)LimitlimitEvaluates the LIMIT clause.MaterializesaveMaterialize rows for input to nested loop joins and some merge joins. Can operate from disk.--parseUsed to parse textual input data during a load.--projectUsed to rearrange columns and compute expressions, that is, project data.Result--Run scalar functions that do not involve any table access.--returnReturn rows to the leader or client.Subplan--Used for certain subqueries.UniqueuniqueEliminates duplicates from SELECT DISTINCT and UNION queries.WindowwindowCompute aggregate and ranking window functions. Can operate from disk.Network Operations:Network (Broadcast)bcastBroadcast is also an attribute of Join Explain operators and steps.Network (Distribute)distDistribute rows to compute nodes for parallel processing by data warehouse cluster.Network (Send to Leader)returnSends results back to the leader for further processing.DML Operations (operators that modify data):Insert (using Result)insertInserts data.Delete (Scan + Filter)deleteDeletes data. Can operate from disk.Update (Scan + Filter)delete, insertImplemented as delete and Insert.Customer D11100Customer B 2 2 200 Customer C 3 3 300 Xia D4 4 400 Kris Du 5 5 500 Customer A 1 6 610 Customer E 2 7 720 Customer F 6 8 860 Customer G 7 9 970 Customer H8 10 1080 Customer D11100Customer E27720Xia D44400Kris Du55500Customer G79970 iPad 1iPad 2iPad 3Definition Size (MB)555Data Size (MB)102525Rendering (MB)6060200Max. Data Slice Size (MB)505050CodeDescription7No certificates available. The SSL certificate was not found. Please confirm the paths used for certificates and file access permissions. 8Certificate validation error. An error is detected while validating a certificate. This error can occur if a root CA certificate is not found in the key database. 9Certificate is not yet valid. This error means that the issuance date of the certificate provided is in the future, so clients will not treat it as valid.10Certificate has expired. This error means that at least one of the certificates provided has expired.11LDAP processing error. An error is detected while setting up the LDAP environment or retrieving an LDAP directory entry.21Unable to verify the first certificate. This error indicates that there is a problem with the certificate chain, and openssl cannot verify that the certificate presented by the device should be trusted.109No certification authority certificates. The key database does not contain any valid certification authority certificates. The SSL run time needs at least one CA or self-signed certificate to perform client authentication. 1GB Device2GB Device4GB DeviceDefinition Size(MB)101010Runtime Size(MB)350 (baseline)7001400Max. Data Slice Size(MB)50100200 Pattern Matching Keyword Matching Supports datasets with more than 500,000 unique keywords. Supports datasets up to 500,000 unique keywords. HyperWeb and HyperIntelligence for Office looks for the unique character pattern in the keywords and surfaces cards based on that criteria. HyperWeb and HyperIntelligence for Office looks for an exact match of the keyword that is stored in the dataset. Card keywords are not downloaded (or cached) in the browser storage. Card keywords are downloaded (or cached) in the browser storage. You will not be able to use the search functionality to explore pattern enabled card sets You will be able to use the search functionality to explore non-pattern enabled card sets. LEVEL_CRITICAL_ERROR Complete system failure, EX. The inability of the Narrowcast Server system to function. The failure of a sole running Distribution Manager (no backup), loss of connection to the repository. (Default – 1)LEVEL_SEVERE_ERRORHigh level system failure, EX. Connection to Subscription Book Module is not established, Delivery Engine failed, Distribution Manager failed, No DNS on a machine (generally errors pertaining to an entire service) (Default – 1)LEVEL_MODERATE_ERRORMid-level system error, EX. Information Source Manager, Information Transmission Module, Document Formatter Module failure (generally errors pertaining to multiple recipients, such as at the segment level) (Default – 1)LEVEL_MINOR_ERRORLowest level system error. EX. Unable to send an email to a recipient, failure of an information object for a recipient (generally errors pertaining to a single recipient) (Default – 1)LEVEL_STAT_EXECUTIONService level tracking information (Default – 1)LEVEL_STAT_SEGMENTSegment level tracking information (Default – 1)LEVEL_STAT_RECIPIENTRecipient level tracking information (Default – 1)LEVEL_DEBUGVery low level diagnostic info used internally by Technology to analyze issues during development or where very detailed tracing information is needed to help diagnose any potential problems. Some components generate vast amounts of LEVEL_DEBUG information. (Default – 0)LEVEL_INFOInformation relating to the general operation of Narrowcast Server. Any components that do not have their own specific Level_INFO_xxxx level will log at this level (such as the SMTP Service) (Default – 1)LEVEL_INFO_DFMGeneral operational messages from the Document Formatter Module (Default – 1)LEVEL_INFO_DMGeneral operational messages from the Distribution Manager EX. System Startup/Shutdown, high level notification of service execution (Default – 1)LEVEL_INFO_EEGeneral operational messages from the Execution Engine (Default – 1)LEVEL_INFO_ERGeneral operational messages from the Execution Recorder (Default – 1)LEVEL_INFO_ISMGeneral operational messages from the Information Source Module (Default – 1)LEVEL_INFO_ITMGeneral operational messages from the Information Transmission Module (Default – 1)LEVEL_INFO_SBMGeneral operational messages from the Subscription Book Module (Default – 1)LEVEL_INFO_SEGeneral operational messages from the Segment Executor (Default – 1)LEVEL_TRACELow level functional tracing information. Any components that do not have their own specific LEVEL_TRACE_xxxx will log at this level (Default – 0)LEVEL_TRACE_DFMLow level tracing information for the Document Formatter Module (Default – 0)LEVEL_TRACE_DMLow level tracing information for the Distribution Manager (Default – 0)LEVEL_TRACE_EELow level tracing information for the Execution Engine (Default – 0)LEVEL_TRACE_ERLow level tracing information for the Execution Recorder (Default – 0)LEVEL_TRACE_ISMLow level tracing information for the Information Source Module (Default – 0)LEVEL_TRACE_ITMLow level tracing information for the Information Transmission Module (Default – 0)LEVEL_TRACE_SBMLow level tracing information for the Subscription Book Module (Default – 0)LEVEL_TRACE_SCHLow level tracing information for the Scheduler – both the Distribution Manager and the GUI will generate messages at this level (the GUI uses the same Scheduler DLL as the Distribution Manager) – details Schedule construction, parameters passed to the scheduler from the repository as well as general execution flow details of the scheduler. Includes both local and UTC dates and times for most operations. (Default – 0)LEVEL_TRACE_SELow level tracing information for the Segment Executor (Default – 0)LEVEL_TRACE_SQLLow level SQL tracing generated by the Narrowcast Server APIs (Default – 0)LEVEL_TRX_RECORDERLow level Transmission Recorder tracing information (Default – 0)DataSQL |
Error in saving the cube, com.microstrategy.webapi.MSTRWebAPIException:() Hortonworks HDP 2.0 Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx Certified Cloudera CDH 5.0 and 5.1 Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx Certified MapR 3.x and 4.x Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx Certified Pivotal HD 2.0.1 and 2.1.0 Strategy ODBC driver for Apache Hive Wire Protocol version 7.10.3.xxx CertifiedBrowseDenyReadGrantWriteDefaultDeleteDefaultControlDefaultUseGrantExecuteGrantEXPLAIN OperatorsQuery Execution StepsDescriptionSCANSequential ScanscanAmazon Redshift relation scan or table scan operator or step. Scans whole table sequentially from beginning to end; also evaluates query constraints for every row (Filter) if specified with WHERE clause. Also used to run INSERT, UPDATE, and DELETE statements.JOINS: Amazon Redshift uses different join operators based on the physical design of the tables being joined, the location of the data required for the join, and specific attributes of the query itself. Subquery Scan -- Subquery scan and append are used to run UNION queries.Nested LoopnloopLeast optimal join; mainly used for cross-joins (Cartesian products; without a join condition) and some inequality joins.Hash JoinhjoinAlso used for inner joins and left and right outer joins and typically faster than a nested loop join. Hash Join reads the outer table, hashes the joining column, and finds matches in the inner hash table. Step can spill to disk. (Inner input of hjoin is hash step which can be disk-based.)Merge JoinmjoinAlso used for inner joins and outer joins (for join tables that are both distributed and sorted on the joining columns). Typically the fastest Amazon Redshift join algorithm, not including other cost considerations.AGGREGATION: Operators and steps used for queries that involve aggregate functions and GROUP BY operations.AggregateaggrOperator/step for scalar aggregate functions.HashAggregateaggrOperator/step for grouped aggregate functions. Can operate from disk by virtue of hash table spilling to disk.GroupAggregateaggrOperator sometimes chosen for grouped aggregate queries if the Amazon Redshift configuration setting for force_hash_grouping setting is off.SORT: Operators and steps used when queries have to sort or merge result sets.SortsortSort performs the sorting specified by the ORDER BY clause as well as other operations such as UNIONs and joins. Can operate from disk.MergemergeProduces final sorted results of a query based on intermediate sorted results derived from operations performed in parallel.EXCEPT, INTERSECT, and UNION operations:SetOp Except [Distinct]hjoinUsed for EXCEPT queries. Can operate from disk based on virtue of fact that input hash can be disk-based.Hash Intersect [Distinct]hjoinUsed for INTERSECT queries. Can operate from disk based on virtue of fact that input hash can be disk-based.Append [All |Distinct]saveAppend used with Subquery Scan to implement UNION and UNION ALL queries. Can operate from disk based on virtue of "save".Miscellaneous/Other:HashhashUsed for inner joins and left and right outer joins (provides input to a hash join). The Hash operator creates the hash table for the inner table of a join. (The inner table is the table that is checked for matches and, in a join of two tables, is usually the smaller of the two.)LimitlimitEvaluates the LIMIT clause.MaterializesaveMaterialize rows for input to nested loop joins and some merge joins. Can operate from disk.--parseUsed to parse textual input data during a load.--projectUsed to rearrange columns and compute expressions, that is, project data.Result--Run scalar functions that do not involve any table access.--returnReturn rows to the leader or client.Subplan--Used for certain subqueries.UniqueuniqueEliminates duplicates from SELECT DISTINCT and UNION queries.WindowwindowCompute aggregate and ranking window functions. Can operate from disk.Network Operations:Network (Broadcast)bcastBroadcast is also an attribute of Join Explain operators and steps.Network (Distribute)distDistribute rows to compute nodes for parallel processing by data warehouse cluster.Network (Send to Leader)returnSends results back to the leader for further processing.DML Operations (operators that modify data):Insert (using Result)insertInserts data.Delete (Scan + Filter)deleteDeletes data. Can operate from disk.Update (Scan + Filter)delete, insertImplemented as delete and Insert.Customer D11100Customer B 2 2 200 Customer C 3 3 300 Xia D4 4 400 Kris Du 5 5 500 Customer A 1 6 610 Customer E 2 7 720 Customer F 6 8 860 Customer G 7 9 970 Customer H8 10 1080 Customer D11100Customer E27720Xia D44400Kris Du55500Customer G79970 iPad 1iPad 2iPad 3Definition Size (MB)555Data Size (MB)102525Rendering (MB)6060200Max. Data Slice Size (MB)505050CodeDescription7No certificates available. The SSL certificate was not found. Please confirm the paths used for certificates and file access permissions. 8Certificate validation error. An error is detected while validating a certificate. This error can occur if a root CA certificate is not found in the key database. 9Certificate is not yet valid. This error means that the issuance date of the certificate provided is in the future, so clients will not treat it as valid.10Certificate has expired. This error means that at least one of the certificates provided has expired.11LDAP processing error. An error is detected while setting up the LDAP environment or retrieving an LDAP directory entry.21Unable to verify the first certificate. This error indicates that there is a problem with the certificate chain, and openssl cannot verify that the certificate presented by the device should be trusted.109No certification authority certificates. The key database does not contain any valid certification authority certificates. The SSL run time needs at least one CA or self-signed certificate to perform client authentication. 1GB Device2GB Device4GB DeviceDefinition Size(MB)101010Runtime Size(MB)350 (baseline)7001400Max. Data Slice Size(MB)50100200 Pattern Matching Keyword Matching Supports datasets with more than 500,000 unique keywords. Supports datasets up to 500,000 unique keywords. HyperWeb and HyperIntelligence for Office looks for the unique character pattern in the keywords and surfaces cards based on that criteria. HyperWeb and HyperIntelligence for Office looks for an exact match of the keyword that is stored in the dataset. Card keywords are not downloaded (or cached) in the browser storage. Card keywords are downloaded (or cached) in the browser storage. You will not be able to use the search functionality to explore pattern enabled card sets You will be able to use the search functionality to explore non-pattern enabled card sets. LEVEL_CRITICAL_ERROR Complete system failure, EX. The inability of the Narrowcast Server system to function. The failure of a sole running Distribution Manager (no backup), loss of connection to the repository. (Default – 1)LEVEL_SEVERE_ERRORHigh level system failure, EX. Connection to Subscription Book Module is not established, Delivery Engine failed, Distribution Manager failed, No DNS on a machine (generally errors pertaining to an entire service) (Default – 1)LEVEL_MODERATE_ERRORMid-level system error, EX. Information Source Manager, Information Transmission Module, Document Formatter Module failure (generally errors pertaining to multiple recipients, such as at the segment level) (Default – 1)LEVEL_MINOR_ERRORLowest level system error. EX. Unable to send an email to a recipient, failure of an information object for a recipient (generally errors pertaining to a single recipient) (Default – 1)LEVEL_STAT_EXECUTIONService level tracking information (Default – 1)LEVEL_STAT_SEGMENTSegment level tracking information (Default – 1)LEVEL_STAT_RECIPIENTRecipient level tracking information (Default – 1)LEVEL_DEBUGVery low level diagnostic info used internally by Technology to analyze issues during development or where very detailed tracing information is needed to help diagnose any potential problems. Some components generate vast amounts of LEVEL_DEBUG information. (Default – 0)LEVEL_INFOInformation relating to the general operation of Narrowcast Server. Any components that do not have their own specific Level_INFO_xxxx level will log at this level (such as the SMTP Service) (Default – 1)LEVEL_INFO_DFMGeneral operational messages from the Document Formatter Module (Default – 1)LEVEL_INFO_DMGeneral operational messages from the Distribution Manager EX. System Startup/Shutdown, high level notification of service execution (Default – 1)LEVEL_INFO_EEGeneral operational messages from the Execution Engine (Default – 1)LEVEL_INFO_ERGeneral operational messages from the Execution Recorder (Default – 1)LEVEL_INFO_ISMGeneral operational messages from the Information Source Module (Default – 1)LEVEL_INFO_ITMGeneral operational messages from the Information Transmission Module (Default – 1)LEVEL_INFO_SBMGeneral operational messages from the Subscription Book Module (Default – 1)LEVEL_INFO_SEGeneral operational messages from the Segment Executor (Default – 1)LEVEL_TRACELow level functional tracing information. Any components that do not have their own specific LEVEL_TRACE_xxxx will log at this level (Default – 0)LEVEL_TRACE_DFMLow level tracing information for the Document Formatter Module (Default – 0)LEVEL_TRACE_DMLow level tracing information for the Distribution Manager (Default – 0)LEVEL_TRACE_EELow level tracing information for the Execution Engine (Default – 0)LEVEL_TRACE_ERLow level tracing information for the Execution Recorder (Default – 0)LEVEL_TRACE_ISMLow level tracing information for the Information Source Module (Default – 0)LEVEL_TRACE_ITMLow level tracing information for the Information Transmission Module (Default – 0)LEVEL_TRACE_SBMLow level tracing information for the Subscription Book Module (Default – 0)LEVEL_TRACE_SCHLow level tracing information for the Scheduler – both the Distribution Manager and the GUI will generate messages at this level (the GUI uses the same Scheduler DLL as the Distribution Manager) – details Schedule construction, parameters passed to the scheduler from the repository as well as general execution flow details of the scheduler. Includes both local and UTC dates and times for most operations. (Default – 0)LEVEL_TRACE_SELow level tracing information for the Segment Executor (Default – 0)LEVEL_TRACE_SQLLow level SQL tracing generated by the Narrowcast Server APIs (Default – 0)LEVEL_TRX_RECORDERLow level Transmission Recorder tracing information (Default – 0)DataSQL |
| select a12.Call_Ctr_Id Call_Ctr_Id, The Engine generates SQL against the ORDER_FACT table and retrieves the SUM(ORDER_AMT). The Average subtotal is calculated by the Analytical Engine and shows the correct results. |
If the Employee attribute is removed and the metric is replaced by , then the following results are returned:
The Engine generates SQL against the ORDER_FACT table and retrieves the SUM(ORDER_AMT). The Average subtotal is calculated by the Analytical Engine and shows the correct results. | The Engine generates SQL against the ORDER_FACT table and retrieves the SUM(ORDER_AMT). The Average subtotal is calculated by the Analytical Engine and shows the correct results. |
| select a12.Call_Ctr_Id Call_Ctr_Id, The Engine generates SQL against the ORDER_FACT table and retrieves the AVG(ORDER_AMT). This is correct but the report will be returning a result that may not be the desired one. This report returns the average 'transaction' sale for each . Under this context, each 'transaction' is identified by the lowest level in the table (): |
SOLUTION:
The Strategy Engine has no way to know what the desired level of aggregation for the report is, so users have to explicitly define the dimensionality of the metric using a nested aggregation.
Nested aggregation is a powerful feature that allows applying aggregation functions to previously aggregated metrics. In this way, more complex metrics can be created without the need of schema modifications or the addition of new tables/views to the project.
If a report that shows and the average of sales is needed, then the following metric is needed:
Avg(Sum() {~+, Employee+} ) {~+}
Notice that the inner function is a sum of with dimensionality on , with standard filtering and grouping. The outer function calculates the average of the inner function at the report level.
IMPORTANT NOTE:
Nested aggregation can also be performed using aggregation functions applied to metrics.
A report that has and will return:
|
|
| select a11.EMP_ID EMP_ID, The Engine generates multi-pass SQL against the ORDER_FACT table and first retrieves the SUM(ORDER_AMT) at the level. Then, in the second pass, it generates SQL against the table created in the fist pas and calculates the average of the sum at the level. This report returns the average sale for each . Notice that the results are exactly the same as the ones shown on the first example that are calculated by the Analytical Engine. |
The same type of analysis can be carried over to another level. If a report that shows and the average sales is requested, then the following metric is needed:
Avg(Sum() {~+, Region+} ) {~+}
This report returns the average sale for each . Notice that the results are exactly the same as the ones shown on the first example that are calculated by the Analytical Engine. | This report returns the average sale for each . Notice that the results are exactly the same as the ones shown on the first example that are calculated by the Analytical Engine. |
| select a11.EMP_ID EMP_ID, The last pass that joins the intermediate results for the final display. The Engine generates multi-pass SQL against the ORDER_FACT table and first retrieves the SUM(ORDER_AMT) at the level. This report returns the average and sale for each . So we can see that the employees with the biggest average sales are in 'France' but the strongest regions are in 'USA'. |
KB419891