Correct Answer: E

The Root Mean Squared Error (RMSE) is a standard and widely used metric for evaluating the accuracy of regression models. The statement that it is invalid is incorrect. Here's a breakdown of why the other statements are or are not valid:

Transformations and RMSE Calculation:If the model predictions were transformed (e.g., using log), they should be converted back to their original scale before calculating RMSE to ensure accuracy in the evaluation. Missteps in this

conversion process can lead to misleading RMSE values. Accuracy of Models:Without additional information, we can't definitively say which model is more accurate without considering their RMSE values properly scaled back to the original

price scale.

Appropriateness of RMSE:RMSE is entirely valid for regression problems as it provides a measure of how accurately a model predicts the outcome, expressed in the same units as the dependent variable.

References:

"Applied Predictive Modeling" by Max Kuhn and Kjell Johnson (Springer, 2013), particularly the chapters discussing model evaluation metrics.

Correct Answer: B

In Spark ML, the linear regression model expects the feature column to be a vector type. However, if thefeaturescolumn in the DataFrametrain_dfis not already in this format (such as being a column of type UDT or a non-vectorized type), the engineerneeds to convert it to a vector column using a transformer likeVectorAssembler. This is a critical step in preparing the data for modeling as Spark ML models require input features to be combined into a single vector column. References: Spark MLlib documentation forLinearRegression:https://spark.apache.org/docs/latest/ml-classification-regression.html#linear-regression

Correct Answer: A

Thesummary()function in PySpark's DataFrame API provides descriptive statistics which include count, mean, standard deviation, min, max, and quantiles for numeric columns. Here are the steps on how it can be used:

Import PySpark:Ensure PySpark is installed and correctly configured in the Databricks environment.

Load Data:Load the data into a Spark DataFrame.

Apply Summary:Usespark_df.summary()to generate summary statistics. View Results:The output from thesummary()function includes the statistics specified in the query (count, mean, standard deviation, min, max, and potentially quartiles

which approximate the interquartile range).

References:

PySpark

Documentation:https://spark.apache.org/docs/latest/api/python/reference/api/pysp ark.sql.DataFrame.summary.html

Correct Answer: B

Vectorized pandas UDFs, also known as Pandas UDFs, are a powerful feature in PySpark that allows for more efficient operations than standard UDFs. They operate by processing data in batches, utilizing vectorized operations that leverage pandas to perform operations on whole batches of data at once. This approach is much moreefficient than processing data row by row as is typical with standard PySpark UDFs, which can significantly speed up the computation. References: PySpark Documentation on UDFs:https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_panda s.html#pandas-udfs-a-k-a-vectorized-udfs

Correct Answer: E

When the goal is to maximize the identification of positive cases in a classification task, the metric of interest isRecall. Recall, also known as sensitivity, measures the proportion of actual positives that are correctly identified by the model (i.e.,

the true positive rate). It is crucial for scenarios where missing a positive case (false negative) has serious implications, such as in medical diagnostics. The other metrics like Precision, RMSE, and Accuracy serve different aspects of

performance measurement and are not specifically focused on maximizing the detection of positive cases alone.

References:

Classification Metrics in Machine Learning (Understanding Recall).

Correct Answer: D

When using theHyperoptlibrary withfmin, the goal is to find the minimum of the objective function. Since you are usingcross_val_scoreto calculate the R2 score which is a measure of the proportion of the variance for a dependent variable

that's explained by an independent variable(s) in a regression model, higher values are better. However,fmin seeks to minimize the objective function, so to align withfmin's goal, you should return the negative of the R2 score (-r2). This way,

by minimizing the negative R2,fminis effectively maximizing the R2 score, which can lead to a more accurate model.

References:

Hyperopt Documentation: http://hyperopt.github.io/hyperopt/ Scikit-Learn documentation on model evaluation: https://scikit-learn.org/stable/modules/model_evaluation.html

Correct Answer: D

To speed up the model tuning process when dealing with a large number of model configurations, parallelizing the hyperparameter search using Hyperopt is an effective approach. Hyperopt provides tools likeSparkTrialswhich can run

hyperparameter optimization in parallel across a Spark cluster.

Example:

fromhyperoptimportfmin, tpe, hp, SparkTrials search_space = {'x': hp.uniform('x',0,1),'y':

hp.uniform('y',0,1) }defobjective(params):returnparams['x'] **2+ params['y'] **2spark_trials = SparkTrials(parallelism=4) best = fmin(fn=objective, space=search_space, algo=tpe.suggest, max_evals=100, trials=spark_trials) References:

Hyperopt Documentation

Correct Answer: C

Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging is a technique that involves training multiple base models (such as decision trees) on different subsets of the data and then combining their predictions to improve overall model performance. Each subset is created by randomly sampling with replacement from the original dataset. The Random Forest algorithm builds multiple decision trees and merges them to get a more accurate and stable prediction. References: Databricks documentation on Random Forest: Random Forest in Spark ML

Correct Answer: A

The code block provided by the machine learning engineer will perform the desired inference when the Feature Store feature set was logged with the model at model_uri. This ensures that all necessary feature transformations and metadata

are available for the model to make predictions. The Feature Store in Databricks allows for seamless integration of features and models, ensuring that the required features are correctly used during inference.

References:

Databricks documentation on Feature Store: Feature Store in Databricks

Correct Answer: C

To filter rows in a Spark DataFrame based on a condition, thefiltermethod is used. In this case, the condition is that the value in the "discount" column should be less than or equal to 0. The correct syntax uses thefiltermethod along with

thecolfunction from pyspark.sql.functions.

Correct code:

frompyspark.sql.functionsimportcol filtered_df = spark_df.filter(col("discount") <=0) Option A and D use Pandas syntax, which is not applicable in PySpark. Option B is closer but misses the use of thecolfunction.

References:

PySpark SQL Documentation