Result

For this example, we want 2 layers

foreground on black with alpha channel

shadow on full white, to multiply on background plate

Render Setup

Render layers are created in Render Setup

For our basic example shot, the following guide will only create the two layers

A more detailed introduction to Render Setup here at Maya 2017 help

Open Render Setup

In the toolbar click on the Render Setup icon

or menu Windows > Rendering Editors > Render Setup

Add Layer and Collection

Add new layer, then right-click to Create Collection

FG Layer

Add objects which should be visible to this Collection

Select the objects in Outliner or viewer

In Render Setup Property Editor, Add them to the Collection

Shadow Layer

Create a second layer for the Shadow, with a new Collection and add the floor object

We need the FG objects to cast shadows on the floor, but being invisible

Copy the FG_objects Collection from the FG layer with copy-paste from Edit menu

Visible Layer

Make sure the Layer’s Visibility is ON, else you won’t see the option on the next step

Create Absolute Override for Visible Layer

Select one of the FG objects (here Sphere1) in Outliner or viewer

In the Shape Attributes > Render Stats > Primary Visibility

right-click and
Create Absolute Override for Visible Layer

Alternative to create Override

Alternatively, you can middle-mouse-drag attribute properties, i.e. the Primary Visibility, to the collection’s drag area

Primary Visibility Override in Render Setup

An Absolute Override is created for the Collection

Untick the Primary Visibility in the Property Editor

Material Override

Right-click on Floor Collection and Create Material Override

Pick aiAmbientOcclusion

Test Render

The occlusion shadows are fine but we want a full-white background for the Shadow layer and a black background for the FG layer

Background Color

Render Settings > Arnold Renderer > Environment > Background

Out Color checkerboard box and Create RaySwitch Shader

aiRaySwitch Camera Attributes

Make sure the Shadow Layer’s Visibility is ON

aiRaySwitch Attributes > Ray Type > Camera > right-click and Create Absolute Override for Visible Layer

aiRaySwitch Override in Render Setup

In the Property Editor, change the color of the Camera to white

Renderable for Batch Rendering

OFF for Scene Layer (masterLayer)

ON for FG and SHADOW layers

This will enable rendering of both layers in one go

Render

Render a test of both layers by toggling the Layer’s Visibility

Make sure the FG layer is on black with alpha channel, the Shadow layer on white

Done, both layers are setup